[06:54:56] (EdgeTrafficDrop) firing: 67% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:59:56] (EdgeTrafficDrop) resolved: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [10:13:35] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Suboptimal anycast routing from leaf switches - https://phabricator.wikimedia.org/T302315 (10ayounsi) Current status is that this is virtually solved (removing the last software blocker for drmrs), the CR above will be needed to allow adver... [10:54:27] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp4025.ulsfo.wmnet with OS buster [11:00:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp4025:9331 is unreachable - https://alerts.wikimedia.org [11:01:33] ^^ expected [11:30:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp4025:9331 is unreachable - https://alerts.wikimedia.org [11:41:34] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp4025.ulsfo.wmnet with OS buster c... [11:42:41] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10Vgutierrez) 05In progress→03Resolved envoy instances are currently being reimaged as HAProxy ones. We're cleaning up and pausing the envoyproxy experiment [11:42:53] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) [11:54:02] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp2040.codfw.wmnet with OS buster [12:00:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp2040:9331 is unreachable - https://alerts.wikimedia.org [12:10:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp2040:9331 is unreachable - https://alerts.wikimedia.org [12:11:11] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp2040:9331 is unreachable - https://alerts.wikimedia.org [12:12:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10User-jbond: varnish filtering: should we automatically update public_cloud_nets - https://phabricator.wikimedia.org/T270391 (10Volans) Although it does not do what we need, some logic to download the lists from multiple clouds can be gath... [12:15:58] 10Traffic: Remove component/varnish6 repo reference in Varnish Test Dockerfile - https://phabricator.wikimedia.org/T302579 (10MMandere) [12:16:26] 10Traffic: Remove component/varnish6 repo reference in Varnish Test Dockerfile - https://phabricator.wikimedia.org/T302579 (10MMandere) p:05Triage→03Low [12:22:26] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp2040:9331 is unreachable - https://alerts.wikimedia.org [12:26:11] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp2040:9331 is unreachable - https://alerts.wikimedia.org [12:31:11] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp2040:9331 is unreachable - https://alerts.wikimedia.org [12:34:00] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) [12:38:40] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp2040.codfw.wmnet with OS buster c... [14:04:40] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp5005.eqsin.wmnet with OS buster [14:21:48] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE Observability (FY2021/2022-Q3), 10User-fgiunchedi: blackbox-exporter no icmp replies on prometheus1006 for a few services - https://phabricator.wikimedia.org/T302265 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi With the icmp probes gone I don'... [14:43:57] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp5005:9331 is unreachable - https://alerts.wikimedia.org [14:44:08] ^^ expected [14:58:57] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp5005:9331 is unreachable - https://alerts.wikimedia.org [15:19:33] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp5005.eqsin.wmnet with OS buster c... [15:38:33] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp3063.esams.wmnet with OS buster [15:39:19] 10Traffic, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission cp4031 - https://phabricator.wikimedia.org/T301269 (10Vgutierrez) [15:45:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp3063:9331 is unreachable - https://alerts.wikimedia.org [15:50:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp3063:9331 is unreachable - https://alerts.wikimedia.org [16:05:33] that's me :) [16:11:57] (EdgeTrafficDrop) firing: 65% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org [16:16:57] (EdgeTrafficDrop) resolved: 67% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org [16:21:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp3063:9331 is unreachable - https://alerts.wikimedia.org [16:26:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp3063:9331 is unreachable - https://alerts.wikimedia.org [16:36:31] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp3063.esams.wmnet with OS buster c... [16:37:26] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez)