[00:28:18] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: [Clean up] Redirect m-dot URLs to canonical domains - https://phabricator.wikimedia.org/T405931#11310318 (10Krinkle) The variability of the CDN purge rate means, to quantify the impact we look at a longer period of time (either range totals, o... [06:41:33] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Main Rollout] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11310584 (10Krinkle) >>! In T403510#11292306, @RolandUnger wrote: > The code of https://es.wikivoyage.org/wiki/Med... [08:40:19] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: [Clean up] Redirect m-dot URLs to canonical domains - https://phabricator.wikimedia.org/T405931#11310705 (10Krinkle) 05Open→03Resolved [09:33:31] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11310845 (10fgiunchedi) We have successfully put in service cloudcephosd1050 and cloudcephosd1051 in {T405478} with single-nic, I haven't seen any problem whatsoever with... [09:49:42] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11310972 (10cmooney) >>! In T399180#11310845, @fgiunchedi wrote: > @taavi @Andrew @cmooney what do you think of the above? The plan sounds good. We need to audit and ma... [10:05:48] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE: Nokia OSPF alerts not working - https://phabricator.wikimedia.org/T408378 (10cmooney) 03NEW p:05Triage→03Medium [11:47:07] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE: Nokia OSPF alerts not working - https://phabricator.wikimedia.org/T408378#11311458 (10cmooney) Ok well I fixed the obvious error but the alerts still aren't firing :( [12:54:20] 06Traffic, 10Hiddenparma, 06SRE: FY 25/26 WE 5.4.5: Enforce global rate-limits - https://phabricator.wikimedia.org/T406545#11311628 (10ssingh) [13:56:55] 06Traffic: varnishtests are broken with podman - https://phabricator.wikimedia.org/T408202#11311852 (10Fabfur) >>! In T408202#11307258, @BCornwall wrote: > To be clear, this isn't a regression in recent changes, is it? not sure about that, I usually run that w/ docker, but I assume yes [14:10:31] 06Traffic, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360#11311915 (10Ottomata) From an-launcher1003: `lang=bash $ sudo -u analytics kerberos-run-command analytics spark3-sql ` `lang=sql DROP TABLE `event... [14:10:33] 06Traffic, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360#11311916 (10Ottomata) [14:14:00] I am going to be draining high-traffic[12] and low-traffic in eqiad (one by one) for the reboots [14:17:02] 👍 [14:18:00] oh right you are on on-call [14:18:02] perfect :D [14:19:29] "great" :D [14:40:04] 10netops, 06Infrastructure-Foundations, 07Documentation: The links under "Test IP fragmentation issues" on `wikitech:Reporting a connectivity issue` no longer appear to work - https://phabricator.wikimedia.org/T407505#11312080 (10LSobanski) 05Open→03Resolved a:03LSobanski I removed the section as t... [14:40:45] 10netops, 06Infrastructure-Foundations, 07sre-alert-triage: Alert in need of triage: PeeringBGPDown (instance cr3-eqsin:9804) - https://phabricator.wikimedia.org/T407833#11312085 (10LSobanski) a:03cmooney [14:57:45] hello traffic friends - I have a should-be-a-noop haproxy config change to roll out to A:cp hosts. any concerns if I move ahead with that at some point soon? [14:57:45] also, any specific concerns about rollout pacing for haproxy vs. other cache host services? (e.g., I would typically roll out varnish or ATS changes over ~ 30m) [14:59:27] swfrench-wmf: no concerns and no distinction really, unless a restart of varnish/ATS is involved, which is pretty rare [14:59:55] varnish restarts we space out over 30m but nothing else in a way (ATS preserves the cache on-disk so we don't space out 30m for that for example) [15:00:27] sukhe: ack, thanks! this is "just" the haproxy config change - nothing else [15:02:00] 06Traffic, 06MW-Interfaces-Team, 06serviceops, 07Epic, and 4 others: Rename "limit group" to "limit policy" - https://phabricator.wikimedia.org/T408192#11312157 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert [15:20:45] * swfrench-wmf is doing this now [15:54:58] * swfrench-wmf is done [19:27:22] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T408148#11313485 (10ssingh) 05Open→03Resolved a:03ssingh Sorry this took a while but this should now be resolved. Thanks to Giusep... [19:44:55] 06Traffic: varnishtests are broken with podman - https://phabricator.wikimedia.org/T408202#11313563 (10ssingh) Brett pointed out that the regression was introduced in https://gerrit.wikimedia.org/r/q/I9fab3e43a39456432eb148df91faffba54b1926e. [20:06:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11313615 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest1006.eqiad.wm... [20:53:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11313740 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet... [22:17:00] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia: add new switches in eqiad/codfw to monitoring and make 'active' - https://phabricator.wikimedia.org/T405558#11314270 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1d57495a-c8c9-4142-bb4a-68c98114d4d1) set by cmooney@cumin1003 for 3 d...