[06:48:56] (EdgeTrafficDrop) firing: 67% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:53:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:59:56] (EdgeTrafficDrop) firing: 67% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:04:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [10:58:53] 10Traffic, 10SRE: Remove old and unused libvarnishapi - https://phabricator.wikimedia.org/T300247 (10MMandere) [11:14:17] 10Traffic, 10SRE: Remove old and unused libvarnishapi - https://phabricator.wikimedia.org/T300247 (10elukey) Thanks a lot for working on this. Can we also remove the old lib from apt and puppet? ` root@apt1001:/srv/wikimedia# reprepro lsbycomponent libvarnishapi1 libvarnishapi1 | 5.1.3-1wm11 | stretch-wikimed... [11:29:31] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp4031.ulsfo.wmnet with OS buster [11:34:38] 10Traffic, 10SRE: Remove old and unused libvarnishapi - https://phabricator.wikimedia.org/T300247 (10MMandere) @elukey, thanks for pointing that out. Yes we can do that, considering `libvarnishapi1` was only required when we were using `varnish 5.1.x` of which we no longer use in production, it is safe to hav... [11:35:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp4031:9331 is unreachable - https://alerts.wikimedia.org [11:37:06] ^^ expected [11:50:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp4031:9331 is unreachable - https://alerts.wikimedia.org [12:09:35] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp4031.ulsfo.wmnet with OS buster completed: - cp4031 (**WARN*... [15:44:07] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10Patch-For-Review: Create Generalised blocking stratagy - https://phabricator.wikimedia.org/T270618 (10jbond) just made the below comment on a tickety which i though may be usefull to capture here to give some context as to what we have to... [15:46:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10Patch-For-Review: Create Generalised blocking stratagy - https://phabricator.wikimedia.org/T270618 (10jbond) >The intension of this CR is to slightly role back that decision and exclude the MX hosts from the abuse_nets ferm context rules... [17:27:00] 10Traffic, 10WMF-Legal, 10Performance-Team (Radar), 10Privacy: Consider disabling Chrome Lite pages for Wikipedia on Chrome on mobile with Cache-Control: no-transform - https://phabricator.wikimedia.org/T218618 (10BBlack) Hi all! I tend to agree with @Peter's various rationales that adding `Cache-control:... [17:39:14] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-drmrs: Q3:(Need By: TBD) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) p:05Triage→03High [17:39:23] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-drmrs: Q3:(Need By: TBD) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) [17:54:45] 10netops, 10Infrastructure-Foundations, 10SRE, 10fundraising-tech-ops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10Papaul) I chat with @Jgreen in IRC we will be doing the upgrade next week on Tuesday the 1st at 10:30 CT [18:27:35] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: TBD) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) [18:28:34] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: TBD) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) I've drafted the directions for remote hands, translating the above diagram to a step by step direction for them to rack our routers... [18:40:22] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: TBD) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) [18:51:42] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) [18:52:05] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) a:05RobH→03ayounsi