[07:20:59] 10Traffic, 10Observability-Logging, 10SRE, 10User-ema: varnishmtail metric loss due to mtail not reading from pipe fast enough - https://phabricator.wikimedia.org/T293879 (10ema) [09:25:37] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ayounsi) Thanks! > I take it the main concern here is allocating a public IPv4 address, which is a scarce resource, no? That's one of... [10:30:56] (VarnishTrafficDrop) firing: 67% GET drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/VarnishTrafficDrop - https://grafana.wikimedia.org/d/000000541/varnish-caching-last-week-comparison?viewPanel=5&var-cluster=text&var-site=esams - https://alerts.wikimedia.org [10:35:56] (VarnishTrafficDrop) resolved: 69% GET drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/VarnishTrafficDrop - https://grafana.wikimedia.org/d/000000541/varnish-caching-last-week-comparison?viewPanel=5&var-cluster=text&var-site=esams - https://alerts.wikimedia.org [10:37:21] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10aborrero) a:05aborrero→03ayounsi >>! In T289882#7450242, @ayounsi wrote: > Which means increasing our attack surface as well as SR... [12:53:30] 10Traffic, 10Observability-Logging, 10SRE, 10Patch-For-Review, 10User-ema: varnishmtail metric loss due to mtail not reading from pipe fast enough - https://phabricator.wikimedia.org/T293879 (10ema) >>! In T293879#7450109, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations),... [13:23:20] 10netops, 10Infrastructure-Foundations: Validate EVPN/VXLAN configuration for Juniper QFX Platform - https://phabricator.wikimedia.org/T294115 (10cmooney) p:05Triage→03Medium [13:25:21] 10Traffic, 10Beta-Cluster-Infrastructure: Varnish reload failing on deployment-cache-upload06 - https://phabricator.wikimedia.org/T294116 (10Majavah) [13:47:11] 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE: Varnish reload failing on deployment-cache-upload06 - https://phabricator.wikimedia.org/T294116 (10ema) 05Open→03Resolved a:03ema I upgraded varnish to 6.0.8 everywhere (see T292290) and forgot about restarting the service on deployment-cache-upload06. I... [18:24:43] 10Traffic, 10Platform Engineering, 10SRE, 10Wikimedia-production-error: Wikimedia\Assert\PostconditionException: Postcondition failed: makeTitleSafe() should always return a Title for the text returned by getRootText(). - https://phabricator.wikimedia.org/T290194 (10Umherirrender) It is possible to get the... [20:02:16] puppet fails on 4 dns hosts, haven't looked at details yet https://puppetboard.wikimedia.org/nodes?status=failed [20:02:44] was looking at puppetboard itself because that alerted in icinga.. there is always another level [20:56:56] (VarnishTrafficDrop) firing: 56% GET drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/VarnishTrafficDrop - https://grafana.wikimedia.org/d/000000541/varnish-caching-last-week-comparison?viewPanel=5&var-cluster=text&var-site=eqiad - https://alerts.wikimedia.org [21:11:56] (VarnishTrafficDrop) resolved: 68% GET drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/VarnishTrafficDrop - https://grafana.wikimedia.org/d/000000541/varnish-caching-last-week-comparison?viewPanel=5&var-cluster=text&var-site=eqiad - https://alerts.wikimedia.org [21:23:41] (VarnishTrafficDrop) firing: (2) 54% GET drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/VarnishTrafficDrop - https://alerts.wikimedia.org [21:43:41] (VarnishTrafficDrop) resolved: 63% GET drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/VarnishTrafficDrop - https://grafana.wikimedia.org/d/000000541/varnish-caching-last-week-comparison?viewPanel=5&var-cluster=text&var-site=codfw - https://alerts.wikimedia.org