[07:00:56] (EdgeTrafficDrop) firing: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:05:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [09:04:06] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar), 10User-ema: Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106 (10ema) Although we did briefly discuss the results of this experiment within #traffic, I don't think we ever publicly disclosed our analysis.... [15:13:17] I'd like to switch a low-traffic service to lvs_setup: https://gerrit.wikimedia.org/r/c/operations/puppet/+/761357 [15:14:26] my understanding is that lvs1020 and lvs2010 are low-traffic backups currently, I would restart pybal on those first [15:17:49] jayme: yeah - there's a bunch of details on that I assume you've read at https://wikitech.wikimedia.org/wiki/LVS#Configure_the_load_balancers [15:17:54] (which aren't perfect, probably!) [15:18:34] but those are the correct backups [15:18:57] bblack: yes, absolutely. Don't know if it's smart to say that beforehand, but this is also not the first lvs-dance for me :) [15:19:33] may the force be with you? :) [15:19:34] or something [15:19:43] I'll take that, thanks :) [18:35:20] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10fkaelin) This would affect the research team, especially if the stat machines are also included in this restriction. Fo... [18:53:39] 10netops, 10Infrastructure-Foundations: Improve Netbox import script to avoid port-number collisions in JunOS - https://phabricator.wikimedia.org/T301392 (10Reedy) [22:33:37] 10Wikimedia-Apache-configuration: Unit tests for apache config/rewrites - https://phabricator.wikimedia.org/T57857 (10RLazarus) a:05RLazarus→03None [22:33:58] 10Wikimedia-Apache-configuration: Unit tests for apache config/rewrites - https://phabricator.wikimedia.org/T57857 (10RLazarus) a:03RLazarus [23:27:35] Watchmouse is talking about an invalid cert for shop.wikimedia.org but ONLY ever when it checks from its "Warsaw, Poland" location. I told it temp. to check every minute and from Spain and Greece and it was always ok. I don't think it's possible we still serve a non-LE cert only in Poland but LE in other countries in Europe, right? [23:28:08] if it would have been Europe vs US I would have been concerned but not "Poland only"