[02:32:23] 10Traffic, 10SRE, 10User-notice: Rate limit requests in violation of User-Agent policy more aggressively - https://phabricator.wikimedia.org/T224891 (10Legoktm) We responded to another set of pages today and most of the offending requests were coming from a public Cloud with no User-agent, so we've banned th... [03:52:59] 10Traffic, 10SRE, 10User-notice: Rate limit requests in violation of User-Agent policy more aggressively - https://phabricator.wikimedia.org/T224891 (10colewhite) Changeset banning empty user agents: https://gerrit.wikimedia.org/r/702027 Result: https://grafana.wikimedia.org/d/000000503/varnish-http-errors?... [08:45:35] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: BGP Policy on aggregate routes prevents them being created in some circumstances. - https://phabricator.wikimedia.org/T283163 (10cmooney) 05Open→03Resolved [09:34:04] apparently I scheduled a meeting during the switchover! [09:34:07] maybe we should not do that? ;) [09:40:05] pretty please :) [10:17:37] are there any switchover things tomorrow? [10:17:41] i haven't seen a time schedule [10:19:34] moved to a slot on thu [10:23:51] is the only viable one? 18:00 CEST on a Thursday is pretty bad for me [10:29:14] pretty much [10:29:18] looks like sukhbir is out all day too [10:29:21] tomorrow at 7pm... [10:30:29] bank holiday on Canada in Thursday IIRC [10:32:31] yeah [10:32:39] otherwise a rare friday meeting could have worked [10:35:53] trying a slot on wed [13:15:56] (VarnishTrafficDrop) firing: 66% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [13:20:56] (VarnishTrafficDrop) firing: (2) 66% GET drop in text@eqiad during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [13:30:57] (VarnishTrafficDrop) resolved: (2) 69% GET drop in text@eqiad during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [14:26:56] (VarnishTrafficDrop) firing: 47% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [14:31:57] (VarnishTrafficDrop) firing: (2) 44% GET drop in text@eqiad during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [14:56:56] (VarnishTrafficDrop) resolved: (2) 65% GET drop in text@eqiad during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org