[05:38:56] (HAProxyEdgeTrafficDrop) firing: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [05:53:56] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:37:29] 10netops, 10Infrastructure-Foundations, 10ops-eqiad: Investigate issue with msw-b7-eqiad - https://phabricator.wikimedia.org/T320598 (10cmooney) p:05Triage→03Medium [08:38:50] 10netops, 10Infrastructure-Foundations, 10ops-eqiad: Investigate issue with msw-b7-eqiad - https://phabricator.wikimedia.org/T320598 (10cmooney) [10:10:56] (HAProxyEdgeTrafficDrop) firing: 49% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [10:15:56] (HAProxyEdgeTrafficDrop) resolved: (3) 60% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [10:48:15] 10Traffic, 10SRE, 10observability: ATS Request Error Ratio SLI shows negative values - https://phabricator.wikimedia.org/T320615 (10Vgutierrez) [10:48:27] 10Traffic, 10SRE, 10observability: ATS Request Error Ratio SLI shows negative values - https://phabricator.wikimedia.org/T320615 (10Vgutierrez) p:05Triage→03Medium [13:55:05] 10netops, 10Infrastructure-Foundations: Add Dell switches support to Homer/Cookbooks - https://phabricator.wikimedia.org/T320638 (10ayounsi) p:05Triage→03Medium [14:18:38] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Investigate issue with msw-b7-eqiad - https://phabricator.wikimedia.org/T320598 (10cmooney) 05Open→03Resolved a:03cmooney @Jclark-ctr has replaced the switch and devices are now back online: `lines=10 cmooney@msw1-eqiad> show ethernet-switch... [15:50:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10Papaul) @Jclark-ctr @Cmjohnson I am planning on moving all the links on cr[1-2]-eqaid from fpc4 to fpc3 for the once in both cr1-eqiad from FPC4 to FPC3 and cr2... [16:00:04] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10Andrew) Let's back off of this plan for OSDs. The two nics on hypervisors are control plane and data plane, whereas on the OSDs they're both dataplane (on... [16:34:00] 10Traffic, 10SRE, 10observability: ATS Request Error Ratio SLI shows negative values - https://phabricator.wikimedia.org/T320615 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez {F35564906} [16:49:45] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo, 10Patch-For-Review: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10RobH) [17:05:34] 10Traffic, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: add HBA355i support to installer - https://phabricator.wikimedia.org/T319067 (10ssingh) Thanks to @MoritzMuehlenhoff and @Volans for their help in resolving the buster Linux 5.10 issue! ` sukhe@cp4045:~$ uname -r 5.10.0-0.deb10.17-amd... [18:27:03] 10Traffic, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: add HBA355i support to installer - https://phabricator.wikimedia.org/T319067 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4045.ulsfo.wmnet with OS buster [18:40:33] 10Traffic, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: add HBA355i support to installer - https://phabricator.wikimedia.org/T319067 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4045.ulsfo.wmnet with OS buster executed with errors: - cp40... [18:42:01] 10Traffic, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: add HBA355i support to installer - https://phabricator.wikimedia.org/T319067 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4045.ulsfo.wmnet with OS buster [19:15:55] 10Traffic, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: add HBA355i support to installer - https://phabricator.wikimedia.org/T319067 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4045.ulsfo.wmnet with OS buster executed with errors: - cp40... [19:16:23] 10Traffic, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: add HBA355i support to installer - https://phabricator.wikimedia.org/T319067 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4045.ulsfo.wmnet with OS buster [19:29:09] 10Traffic, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: add HBA355i support to installer - https://phabricator.wikimedia.org/T319067 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4045.ulsfo.wmnet with OS buster executed with errors: - cp40...