[08:34:42] 10netops, 10Infrastructure-Foundations: cr2-esams:FPC0 Parity error - https://phabricator.wikimedia.org/T318783 (10ayounsi) p:05Triage→03Medium [08:46:29] vgutierrez: please ping when you are about so i can merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/832268 thx [08:48:19] jbond: he is on vacations [08:49:11] oh sneeky i had a meeting with him ytesterday so assumed he would be in this week. ok i can at least merge the first one in that cr set [08:49:21] thanks XioNoX [09:34:22] Sorry about that:) [09:53:55] 10Traffic, 10DNS, 10Infrastructure-Foundations, 10SRE-tools, 10Patch-For-Review: DNS repo: add Jenkins job to ensure there are no duplicates - https://phabricator.wikimedia.org/T155761 (10Volans) I think that the only patch left to be merged is https://gerrit.wikimedia.org/r/c/operations/dns/+/793728, pe... [12:07:11] 10Traffic, 10SRE, 10SRE-OnFire (FY2021/2022-Q4): ncredir redirects for status.wiki* --> status.wikimedia.org - https://phabricator.wikimedia.org/T318804 (10CDanis) [15:20:32] vgu.tierrez: enjoy :) [15:25:04] 10netops, 10Infrastructure-Foundations, 10SRE: Add peering sessions on cr1-eqiad Equinix port - https://phabricator.wikimedia.org/T294948 (10ayounsi) 05Open→03Resolved a:03ayounsi I used the new peering cookbook to mass email the ~76 AS that were on cr2 but not cr1. About ~10 replied quickly and some n... [15:32:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad, 10Sustainability (Incident Followup): eqiad row C switch fabric recabling - https://phabricator.wikimedia.org/T313384 (10ayounsi) @Jclark-ctr @wiki_willy do you have any update about this task? This is set to high priority as it is to prevent an... [20:29:57] (HAProxyEdgeTrafficDrop) firing: 44% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:32:16] (VarnishTrafficDrop) firing: (2) Varnish traffic in eqiad has dropped 51.402989513539175% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [20:34:56] (HAProxyEdgeTrafficDrop) firing: (5) 49% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:37:16] (VarnishTrafficDrop) resolved: (3) Varnish traffic in eqiad has dropped 52.05598501170528% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [20:39:57] (HAProxyEdgeTrafficDrop) firing: (5) 49% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:44:56] (HAProxyEdgeTrafficDrop) resolved: (5) 52% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [23:09:16] (VarnishTrafficDrop) firing: Varnish traffic in esams has dropped 51.29475873578979% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [23:09:56] (HAProxyEdgeTrafficDrop) firing: (5) 14% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [23:14:16] (VarnishTrafficDrop) firing: (11) Varnish traffic in codfw has dropped 26.60709054779857% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [23:14:56] (HAProxyEdgeTrafficDrop) firing: (6) 22% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [23:19:16] (VarnishTrafficDrop) resolved: (12) Varnish traffic in codfw has dropped 27.026310445931212% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [23:19:56] (HAProxyEdgeTrafficDrop) resolved: (6) 22% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop