[02:24:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 64% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[02:29:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: 66% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[06:49:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: (2) 22% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[06:54:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (4) 44% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[07:01:16] <wikibugs>	 10netops, 10Infrastructure-Foundations: cr2-eqiad:FPC3 partial failure (PIC2/3) - https://phabricator.wikimedia.org/T312745 (10ayounsi) p:05Triage→03High
[07:01:53] <wikibugs>	 10netops, 10Infrastructure-Foundations: cr2-eqiad:FPC3 partial failure (PIC2/3) - https://phabricator.wikimedia.org/T312745 (10ayounsi) High severity Case Number 2022-0711-508366
[09:08:20] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10ayounsi) Great, so next step are: # Install the breakout panels, (document them, similar to {T304710}) # Pre-populate the ports/panels that will be used with th...
[09:45:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: (2) 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[09:50:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (2) 67% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[12:14:43] <wikibugs>	 10Traffic, 10Observability-Alerting, 10SRE, 10Patch-For-Review, 10User-fgiunchedi: Migrate Traffic Prometheus alerts from Icinga to Alertmanager - https://phabricator.wikimedia.org/T300723 (10fgiunchedi) >>! In T300723#8066385, @BCornwall wrote: > @fgiunchedi  >  > Looks like the rules mentioned in the t...
[13:24:38] <jinxer-wm>	 (LVSHighRX) firing: Excessive RX traffic on lvs1019:9100 (eno1np0) #page - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs1019 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[13:29:38] <jinxer-wm>	 (LVSHighRX) resolved: Excessive RX traffic on lvs1019:9100 (eno1np0) #page - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs1019 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[13:42:43] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: cr2-eqiad:FPC3 partial failure (PIC2/3) - https://phabricator.wikimedia.org/T312745 (10ayounsi) a:03Cmjohnson Juniper agreed on an RMA, forwarded the email thread to Chris for the shipping details.  @Cmjohnson please sync up with Netops once rec...
[15:20:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 33% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[15:25:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (4) 51% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[19:02:55] <wikibugs>	 10Traffic, 10SRE: HTTP/2 requests fail with too-long URLs - https://phabricator.wikimedia.org/T209590 (10Krinkle)
[20:47:17] <wikibugs>	 10Traffic, 10DNS, 10SRE, 10WMF-Legal, and 4 others: Setup redirect of policy.wikimedia.org to Advocacy portal on Foundation website - https://phabricator.wikimedia.org/T310738 (10Varnent) >>! In T310738#8035453, @Dzahn wrote: >>>! In T310738#8033789, @LSobanski wrote: >> @Varnent After chatting about this...