[04:18:57] (HAProxyEdgeTrafficDrop) firing: 58% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [04:28:57] (HAProxyEdgeTrafficDrop) resolved: 67% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [06:18:56] (HAProxyEdgeTrafficDrop) firing: (3) 44% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [06:23:56] (HAProxyEdgeTrafficDrop) resolved: (5) 49% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [07:50:19] 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Undocumented IP on WMCS network - https://phabricator.wikimedia.org/T315955 (10taavi) >>! In T315955#8188684, @Andrew wrote: > This is good to know! I only recently changed those to CNAMEs, so I'll switch them back when... [08:02:56] (HAProxyEdgeTrafficDrop) firing: 69% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:07:56] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:35:35] 10Traffic, 10Gerrit, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Development services): Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10kostajh) >>! In T191183#6658569, @gerritbot wrote: > Change 456437 **abandoned** by Hashar: > [operations/puppet@production] Gerrit: S... [11:56:01] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade core routers to Junos 20+ - https://phabricator.wikimedia.org/T295690 (10ayounsi) [12:29:44] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade core routers to Junos 20+ - https://phabricator.wikimedia.org/T295690 (10ayounsi) [12:30:42] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10ayounsi) [12:46:55] 10netops, 10Infrastructure-Foundations: Upgrade management routers to Junos 21 - https://phabricator.wikimedia.org/T316529 (10ayounsi) p:05Triage→03Low [13:03:40] 10netops, 10Infrastructure-Foundations: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) p:05Triage→03Low [13:17:25] 10netops, 10Infrastructure-Foundations: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10ayounsi) [13:26:44] 10netops, 10Infrastructure-Foundations: Upgrade network devices - https://phabricator.wikimedia.org/T316539 (10ayounsi) p:05Triage→03Lowest [13:27:49] 10netops, 10Infrastructure-Foundations: Upgrade network devices - https://phabricator.wikimedia.org/T316539 (10ayounsi) [13:27:55] 10netops, 10Infrastructure-Foundations: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) [13:28:01] 10netops, 10Infrastructure-Foundations: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10ayounsi) [13:28:09] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10ayounsi) [13:28:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10fundraising-tech-ops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10ayounsi) [13:28:25] 10netops, 10Infrastructure-Foundations: Upgrade network devices - https://phabricator.wikimedia.org/T316539 (10ayounsi) [13:28:31] 10netops, 10Infrastructure-Foundations: Upgrade network devices to Junos 20+ - https://phabricator.wikimedia.org/T316539 (10ayounsi) [13:41:10] 10netops, 10Infrastructure-Foundations, 10fundraising-tech-ops: Upgrade fasw to Junos 21 - https://phabricator.wikimedia.org/T316542 (10ayounsi) p:05Triage→03Low [13:41:32] 10netops, 10Infrastructure-Foundations, 10fundraising-tech-ops: Upgrade fasw to Junos 21 - https://phabricator.wikimedia.org/T316542 (10ayounsi) [13:41:40] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade network devices to Junos 20+ - https://phabricator.wikimedia.org/T316539 (10ayounsi) [13:45:19] 10netops, 10Infrastructure-Foundations, 10SRE: all network devices must run OpenSSH >= 7.2p1 but != 7.4p1 - https://phabricator.wikimedia.org/T254013 (10ayounsi) [13:45:27] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade network devices to Junos 20+ - https://phabricator.wikimedia.org/T316539 (10ayounsi) [13:46:33] 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10cmooney) p:05Triage→03Low [13:47:37] 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10ayounsi) [13:47:45] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade network devices to Junos 20+ - https://phabricator.wikimedia.org/T316539 (10ayounsi) [13:47:58] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10taavi) [13:58:07] 10Traffic: ATS isn't honoring the cache policy set in cache::alternate_domains on some cases - https://phabricator.wikimedia.org/T316545 (10Vgutierrez) [14:39:47] 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Routing loop for unused WMCS IPs in 185.15.56.0/24 - https://phabricator.wikimedia.org/T315956 (10cmooney) a:03cmooney Yeah I hadn't considered that when I made the change originally. Figured it was a simplification but the routing loop is... [15:21:19] 10Traffic, 10SRE, 10Patch-For-Review: ATS isn't honoring the cache policy set in cache::alternate_domains on some cases - https://phabricator.wikimedia.org/T316545 (10Vgutierrez) [15:21:45] 10Traffic, 10SRE, 10Patch-For-Review: ATS isn't honoring the cache policy set in cache::alternate_domains on some cases - https://phabricator.wikimedia.org/T316545 (10Vgutierrez) p:05Triage→03Medium a:03Vgutierrez [18:12:12] 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Routing loop for unused WMCS IPs in 185.15.56.0/24 - https://phabricator.wikimedia.org/T315956 (10cmooney) I've made these changes now manually. One change to note is the included ASNs of eBGP peers cloudsw1-e4 and cloudsw1-f4 on the aggrega... [18:27:39] Hey folks - I come to you after the meeting we had earlier with kwakuofori, vgutierrez and bblack [18:28:06] We have tracked down our issue to port-80 redirects only, with an exact match on numbers for the specific cases of interest [18:28:34] This means we don't need any further action from you folks :) Many thanks again for your help and support [18:28:58] happy to help :) [18:41:30] nice! [19:04:57] (HAProxyEdgeTrafficDrop) firing: 64% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [19:09:57] (HAProxyEdgeTrafficDrop) resolved: 64% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:22:19] 10Traffic, 10SRE: strip non session cookies before cache lookup in ATS - https://phabricator.wikimedia.org/T316338 (10bd808) >>! In T316338#8193037, @gerritbot wrote: > Change 826785 **merged** by Vgutierrez: > %%%[operations/puppet@production] trafficserver: Hide non session cookies during cache lookup%%% > h... [23:33:03] 10netops, 10Infrastructure-Foundations, 10SRE, 10fundraising-tech-ops: Upgrade fasw to Junos 21 - https://phabricator.wikimedia.org/T316542 (10Dwisehaupt) @ayounsi We have a maintenance week for frack scheduled for Sep 26-30. Would sometime that week be good for you? We could do fasw1-c-codfw before then i... [23:35:09] (HAProxyEdgeTrafficDrop) firing: 48% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [23:39:57] (HAProxyEdgeTrafficDrop) resolved: 60% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop