[01:47:57] (HAProxyEdgeTrafficDrop) firing: 47% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [01:52:56] (HAProxyEdgeTrafficDrop) firing: (5) 56% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [01:57:56] (HAProxyEdgeTrafficDrop) resolved: (5) 56% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:09:58] 10Traffic, 10SRE: Improve handling/logging of HAproxy emergency log messages - https://phabricator.wikimedia.org/T306236 (10Vgutierrez) 05Open→03In progress [08:18:21] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install new mr1-ulsfo - https://phabricator.wikimedia.org/T294314 (10ayounsi) [08:22:39] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install new mr1-ulsfo - https://phabricator.wikimedia.org/T294314 (10ayounsi) a:05ayounsi→03RobH Netbox has been updated to the best of my knowledge using the new https://netbox.wikimedia.org/extras/script... [08:56:35] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) replace mr1-eqiad - https://phabricator.wikimedia.org/T294474 (10ayounsi) Ping? [13:52:14] 10Traffic, 10SRE: Clean up Traffic Grafana dashboards to reflect HA-Proxy metrics - https://phabricator.wikimedia.org/T304153 (10MMandere) 05Open→03In progress [13:52:23] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10MMandere) [14:46:55] 10Traffic, 10SRE, 10Patch-For-Review: Improve handling/logging of HAproxy emergency log messages - https://phabricator.wikimedia.org/T306236 (10Vgutierrez) So I was considering a third approach, parsing the termination_state field from HAProxy request log, but it won't give the exact issue (PC and RC seems t... [17:06:48] 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10dom_walden) This is happening again. I am also seeing: ` Request from 52.225.87.246 via deployment-cache-text06 d... [17:18:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Agree how to handle port-block speeds for QFX5120-48Y - https://phabricator.wikimedia.org/T303529 (10cmooney) So to confirm it the configuration detailed above does not work: ` mooney@cloudsw1-e4-eqiad> show configuration chassis | display... [17:19:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: 2M 25G DAC testing - https://phabricator.wikimedia.org/T306220 (10cmooney) @Jclark-ctr that's great. I've been able to finish off the testing. Feel free to remove those cables and close off this task. Thanks :) [17:20:15] 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10Zabe) ` zabe@deployment-mediawiki12:~$ sudo tail /var/log/apache2.log Apr 19 17:13:55 deployment-mediawiki12 apac... [18:15:45] 10Traffic, 10Performance-Team, 10SRE, 10serviceops: Potential navtiming_responseStart regression as of 13 Mar 2022 - https://phabricator.wikimedia.org/T303782 (10Peter) I'll just check Chrome vs Safari on mobile. When 100 rolled out I saw this https://phabricator.wikimedia.org/T305122#7838322 on WebPageTes... [20:06:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: 2M 25G DAC testing - https://phabricator.wikimedia.org/T306220 (10Jclark-ctr) 05Open→03Resolved a:05cmooney→03Jclark-ctr Thanks Removed both cables closing task [20:09:47] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) replace mr1-eqiad - https://phabricator.wikimedia.org/T294474 (10Jclark-ctr) cable has been removed pinged on irc [20:38:57] (HAProxyEdgeTrafficDrop) firing: 57% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:43:57] (HAProxyEdgeTrafficDrop) resolved: 60% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop