[00:06:56] (EdgeTrafficDrop) firing: (2) 30% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [00:11:56] (EdgeTrafficDrop) resolved: (3) 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [00:38:57] (EdgeTrafficDrop) firing: (2) 61% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [00:43:56] (EdgeTrafficDrop) resolved: (2) 62% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [01:40:11] 10Traffic, 10SRE, 10envoy, 10serviceops, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) [01:46:51] 10Traffic, 10SRE, 10envoy, 10serviceops, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) 1.15.4 is still running in a few places on k8s -- after bumping the default version, I rolled out all services where that was the only diff. Some servic... [01:49:03] 10Traffic, 10SRE, 10envoy, 10serviceops, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) [09:14:23] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp5004.eqsin.wmnet with OS buster [09:20:57] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp5004:9331 is unreachable - https://alerts.wikimedia.org [09:31:56] (EdgeTrafficDrop) firing: 68% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [09:36:56] (EdgeTrafficDrop) resolved: 68% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [09:44:56] (EdgeTrafficDrop) firing: 50% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [10:00:57] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp5004:9331 is unreachable - https://alerts.wikimedia.org [10:04:56] (EdgeTrafficDrop) resolved: 67% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [10:11:26] (EdgeTrafficDrop) firing: (3) 44% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [10:24:50] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp5004.eqsin.wmnet with OS buster c... [10:37:11] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp4024.ulsfo.wmnet with OS buster [10:41:26] (EdgeTrafficDrop) resolved: (2) 67% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [10:43:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp4024:9331 is unreachable - https://alerts.wikimedia.org [10:48:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp4024:9331 is unreachable - https://alerts.wikimedia.org [10:50:25] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp3059.esams.wmnet with OS buster [10:56:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp3059:9331 is unreachable - https://alerts.wikimedia.org [11:01:56] (VarnishPrometheusExporterDown) firing: (2) Varnish Exporter on instance cp3059:9331 is unreachable - https://alerts.wikimedia.org [11:14:52] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp4024.ulsfo.wmnet with OS buster c... [11:16:22] 10Traffic, 10Data-Engineering, 10Event-Platform, 10SRE, and 2 others: Banner sampling leading to a relatively wide site outage (mostly esams) - https://phabricator.wikimedia.org/T303036 (10jcrespo) [11:21:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp3059:9331 is unreachable - https://alerts.wikimedia.org [11:44:22] 10Traffic, 10DNS, 10SRE, 10Wikimedia Enterprise: 301 redirect setup for wikimediaenterprise - https://phabricator.wikimedia.org/T302756 (10Protsack.stephan) p:05Triage→03Low [13:05:35] 10Traffic, 10SRE, 10ops-eqsin: SMART error (CurrentPendingSector) detected on host: cp5004 - https://phabricator.wikimedia.org/T303043 (10JMeybohm) [14:46:48] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp3059.esams.wmnet with OS buster c... [15:16:18] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp2038.codfw.wmnet with OS buster [15:22:13] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp1086.eqiad.wmnet with OS buster [15:22:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp2038:9331 is unreachable - https://alerts.wikimedia.org [15:27:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp2038:9331 is unreachable - https://alerts.wikimedia.org [15:33:11] (VarnishPrometheusExporterDown) firing: (2) Varnish Exporter on instance cp1086:9331 is unreachable - https://alerts.wikimedia.org [15:38:11] (VarnishPrometheusExporterDown) firing: (2) Varnish Exporter on instance cp1086:9331 is unreachable - https://alerts.wikimedia.org [15:43:11] (VarnishPrometheusExporterDown) firing: (2) Varnish Exporter on instance cp1086:9331 is unreachable - https://alerts.wikimedia.org [15:53:11] (VarnishPrometheusExporterDown) firing: (2) Varnish Exporter on instance cp1086:9331 is unreachable - https://alerts.wikimedia.org [15:56:25] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp2038.codfw.wmnet with OS buster c... [15:58:11] (VarnishPrometheusExporterDown) resolved: (2) Varnish Exporter on instance cp1086:9331 is unreachable - https://alerts.wikimedia.org [16:00:08] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp1086.eqiad.wmnet with OS buster c... [16:00:49] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez)