[06:56:56] (EdgeTrafficDrop) firing: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:01:56] (EdgeTrafficDrop) resolved: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:37:40] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10Joe) >> **First**: How difficult & how much overhead would it be to make the proxy redirect requests made to internal d... [07:59:02] is our drmrs setup at the point where we can merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/737328 ? [08:40:26] moritzm: in a few days I'd say, routers should be racked today so a few days to get them all configured [08:50:08] excellent, thanks [11:57:48] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Enable IPv6 for Wikidough - https://phabricator.wikimedia.org/T301165 (10cmooney) @ssingh I haven't had time to go through all of this and work it out, but some things seem clear enough. As per @Majavah's comments on the CR, i... [12:16:36] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Enable IPv6 for Wikidough - https://phabricator.wikimedia.org/T301165 (10ayounsi) Puppet will automatically filter v4 from b6 neighbors and should do the right thing when v4 and v6 IPs are mixed in `neighbors_list` https://git... [13:08:07] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Enable IPv6 for Wikidough - https://phabricator.wikimedia.org/T301165 (10cmooney) Thanks @ayounsi makes sense. I wonder what will happen in this case, with the input being two IPv4 addresses? Will it interpret neighbors_list... [13:48:36] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10Ottomata) @Joe I think most of the usages of webproxy that folks here are concerned with aren't by production services.... [14:04:12] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10akosiaris) >>! In T300977#7710861, @Ottomata wrote: > @Joe I think most of the usages of webproxy that folks here are c... [14:23:58] 10netops, 10Discovery, 10Infrastructure-Foundations, 10SRE: Speed up network connections for Elastic hosts - https://phabricator.wikimedia.org/T301577 (10bking) Hello Cathal, Ryan and I are in training this week, but I would love to meet with you (maybe next week) and talk specifics about network (or any... [14:38:24] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Enable IPv6 for Wikidough - https://phabricator.wikimedia.org/T301165 (10ssingh) >>! In T301165#7710616, @ayounsi wrote: > Puppet will automatically filter v4 from v6 neighbors and should do the right thing when v4 and v6 IPs a... [14:49:03] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Enable IPv6 for Wikidough - https://phabricator.wikimedia.org/T301165 (10cmooney) > Do we anticipate other for uses of the anycast service? I don't think we would rule out introducing new services based on it, but right now I... [15:32:37] 10netops, 10Discovery, 10Infrastructure-Foundations, 10SRE: Speed up network connections for Elastic hosts - https://phabricator.wikimedia.org/T301577 (10cmooney) Thanks Brian > Ryan and I are in training this week, but I would love to meet with you (maybe next week) and talk specifics about network (or a... [20:22:57] (EdgeTrafficDrop) firing: 52% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org [20:27:57] (EdgeTrafficDrop) resolved: 69% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org [20:39:17] 10Traffic, 10SRE, 10Upstream: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required') - https://phabricator.wikimedia.org/T300366 (10TheDJ) Upstream made some changes, but it seems there are some post-merge concerns that came up in https://github.com/envoyproxy/envoy/p...