[08:21:06] 06serviceops, 10MW-on-K8s, 06Traffic, 13Patch-For-Review: Some sites try and fail to serve favicon.ico - https://phabricator.wikimedia.org/T374997#10163389 (10akosiaris) a:03akosiaris @matmarex , I am pretty confident I 've found the reason and have a patch at https://gerrit.wikimedia.org/r/c/operations/... [09:03:03] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 4 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#10163581 (10MoritzMuehlenhoff) [09:37:16] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130#10163695 (10Trizek-WMF) [09:39:33] 06serviceops, 06DBA, 06SRE, 10MediaWiki-Platform-Team (Radar), 10Sustainability (Incident Followup): In the aftermath of T370304: Brainstorming of short- and medium-term observability / quality-of-life production changes - https://phabricator.wikimedia.org/T372943#10163697 (10ABran-WMF) >>! In T372943#10... [09:42:54] 06serviceops, 06Infrastructure-Foundations, 06SRE: confd setup left without configuration doesn't stop confd - https://phabricator.wikimedia.org/T356296#10163701 (10JMeybohm) This came up during {T374366} - currently it is not possible to disable `defs_from_etcd` in a clean way. [09:47:00] 06serviceops, 07Datacenter-Switchover: Verify our current wikikube capacity (in both DCs) can handle all our traffic - https://phabricator.wikimedia.org/T371273#10163713 (10akosiaris) Nicely done! Thanks for the detailed writeup! > (note: it seems odd that that codfw has ~ 1k more allocatable CPU - I've not... [11:46:26] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes: Degraded RAID on wikikube-worker2092 - https://phabricator.wikimedia.org/T374409#10163828 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ae34a38f-1cf6-4321-bfa2-45c744f8ff06) set by cgoubert@cumin1002 for 7 days, 0:00:00 on 1 host... [12:36:11] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130#10163928 (10Trizek-WMF) [12:54:09] 06serviceops, 10Prod-Kubernetes, 03Discovery-Search (Current work), 07Kubernetes, 13Patch-For-Review: Use kafka-main-[eqiad|codfw].external-services.svc.cluster.local to discover kafka brokers in kafka client running in k8s - https://phabricator.wikimedia.org/T374729#10163973 (10JMeybohm) As no better id... [13:09:30] 06serviceops, 06SRE, 13Patch-For-Review: mw2420-mw2451 do have unnecessary raid controllers (configured) - https://phabricator.wikimedia.org/T358489#10164005 (10Clement_Goubert) [14:34:51] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10164170 (10Lucas_Werkmeister_WMDE) I feel like this task is missing an equivalent of {T358286}, is that intentional? For instance, the switchover hasn’t cur... [14:40:22] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10164176 (10Dreamy_Jazz) Noting that I am running an import script for T375203 on a `tmux` session. The tmux session is named `T375203` and should be safe to... [15:00:59] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10164227 (10JMeybohm) [15:29:26] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10164278 (10Scott_French) @Dreamy_Jazz - Thank you for the heads-up about both of the maintenance scripts. That will be quite helpful in tracking down linger... [15:50:03] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10164323 (10Lucas_Werkmeister_WMDE) Alright, thanks for clarifying :) [16:12:03] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10164363 (10Scott_French) [16:23:53] 06serviceops, 07Datacenter-Switchover: Verify our current wikikube capacity (in both DCs) can handle all our traffic - https://phabricator.wikimedia.org/T371273#10164400 (10Scott_French) 05Open→03Resolved Thanks, @akosiaris! That's a good point, and agreed that a different hardware generation should be... [16:43:50] 06serviceops, 07Datacenter-Switchover: sre.discovery.datacenter should handle depooled authdns hosts - https://phabricator.wikimedia.org/T375285 (10Scott_French) 03NEW [16:52:19] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Pre-switchover cookbook testing - https://phabricator.wikimedia.org/T374047#10164504 (10Scott_French) 05Open→03Resolved With those two minor changes merged, I believe there's nothing else explicitly tracked here. I've opened T375285 for the o... [17:40:53] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10164709 (10CDanis) As best I know here's the current state of this task: A couple weeks ago we discussed on -k8s-sig IRC that the "some coredns pod IPs get assigned staticall...