[06:48:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:03:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [09:04:02] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp3062.esams.wmnet with OS buster [09:10:57] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp3062:9331 is unreachable - https://alerts.wikimedia.org [09:11:19] ^^ expected (host being reimaged) [09:20:57] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp3062:9331 is unreachable - https://alerts.wikimedia.org [10:01:55] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp3062.esams.wmnet with OS buster completed: - cp3062 (**WARN*... [10:16:36] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10Vgutierrez) [10:53:54] 10Traffic, 10SRE: Remove old and unused libvarnishapi - https://phabricator.wikimedia.org/T300247 (10ema) We should probably add a `Conflicts: libvarnishapi1` to our varnish 6 packaging, or whatever relationship magic is the right one to ensure that if varnish 6 is installed, libvarnishapi1 is not. [10:55:19] 10Traffic, 10SRE: Remove old and unused libvarnishapi - https://phabricator.wikimedia.org/T300247 (10MoritzMuehlenhoff) >>! In T300247#7667111, @ema wrote: > We should probably add a `Conflicts: libvarnishapi1` to our varnish 6 packaging, or whatever relationship magic is the right one to ensure that if varnis... [11:06:12] bblack, mmandere, FYI netflow6001 is live [11:22:30] niiice [11:35:22] XioNoX: thank you! [12:57:52] 10Traffic, 10SRE, 10Patch-For-Review: Create Ganeti VMs for durum in drmrs - https://phabricator.wikimedia.org/T300158 (10MMandere) [14:39:40] 10Traffic, 10SRE, 10ops-ulsfo: SMART errors on cp4031 - https://phabricator.wikimedia.org/T300493 (10Vgutierrez) p:05Triage→03Medium [14:42:12] 10Traffic, 10SRE, 10ops-ulsfo: SMART errors on cp4031 - https://phabricator.wikimedia.org/T300493 (10Vgutierrez) @RobH just pinging you to make you aware of this one for your next visit to the datacenter [14:46:58] 10Traffic, 10SRE, 10ops-ulsfo: SMART errors on cp4031 - https://phabricator.wikimedia.org/T300493 (10RobH) a:03RobH I'll put in a self dispatch for a new SSD and go swap when it shows up. Will update later today. [16:12:58] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp2039.codfw.wmnet with OS buster [16:19:57] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp2039:9331 is unreachable - https://alerts.wikimedia.org [16:49:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp2039:9331 is unreachable - https://alerts.wikimedia.org [16:58:47] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp2039.codfw.wmnet with OS buster completed: - cp2039 (**WARN*... [17:22:37] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10Vgutierrez) [17:36:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10fundraising-tech-ops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10Papaul) ` request system software add no-copy unlink ` started at 10:38am CT and complete at 10.52am CT ~ 13mins ` request system reboot ` started... [17:39:07] 10netops, 10Infrastructure-Foundations, 10SRE, 10fundraising-tech-ops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10Papaul) 05Open→03Resolved a:03Papaul Complete [17:42:17] 10netops, 10Infrastructure-Foundations, 10SRE, 10fundraising-tech-ops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10Papaul) 05Resolved→03Open [17:53:24] 10netops, 10Infrastructure-Foundations, 10SRE, 10fundraising-tech-ops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10Jgreen) [17:54:45] 10netops, 10Infrastructure-Foundations, 10SRE, 10fundraising-tech-ops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10Jgreen) We're planning to do the upgraded during the Fundraising planned maintenance window of 5/16/2022 to 5/20/2022. [19:55:46] 10Domains, 10Traffic, 10DNS, 10SRE, and 3 others: Project Unseen campaign URL redirect - https://phabricator.wikimedia.org/T300398 (10Varnent) Thank you so very much @Ladsgroup!! :)