[00:53:36] 06serviceops: Turn down unused swift-r[ow] discovery services - https://phabricator.wikimedia.org/T376237 (10Scott_French) 03NEW [00:53:52] 06serviceops: Turn down unused swift-r[ow] discovery services - https://phabricator.wikimedia.org/T376237#10194488 (10Scott_French) p:05Triage→03Low [00:56:30] 06serviceops: Turn down unused swift-r[ow] discovery services - https://phabricator.wikimedia.org/T376237#10194492 (10Scott_French) [00:58:22] 06serviceops: Turn down unused swift-r[ow] discovery services - https://phabricator.wikimedia.org/T376237#10194493 (10Scott_French) [07:57:33] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10194826 (10elukey) It failed with: ` failed to garbage collect: failed to mark: swift: swift: failed to retrieve tags unknown repository name... [11:17:24] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10195436 (10jijiki) [11:18:00] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: Cleanup: Wikitech code leftovers - https://phabricator.wikimedia.org/T371378#10195439 (10jijiki) [12:13:50] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10195571 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestage2001.codfw.wmnet with OS bookworm [13:14:24] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10195834 (10jijiki) @Jclark-ctr very sorry for missing your message. I added the hosts in `site.pp`, there is already a 'mc*' reference in `preseed.yaml` to mat... [14:29:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10196152 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestage2001.codfw.wmnet with OS bookworm com... [14:34:27] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10196171 (10elukey) Backtracking a little before proceeding further, there are some things that I don't fully grasp. In most of the docs that... [14:46:47] 06serviceops, 06Infrastructure-Foundations, 06SRE: Timeout while retrieving the catalog from the Docker Registry - https://phabricator.wikimedia.org/T376285 (10elukey) 03NEW [15:09:02] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10196287 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switcho... [15:13:12] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10196295 (10JMeybohm) kubestage2001 has been repartitioned and reimaged and is now running with containerd. There is one immediate issue though: ` Oc... [15:27:33] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10196359 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in eqiad: Datacenter Switcho... [15:32:53] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10196415 (10cmooney) [16:06:54] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10196688 (10CDanis) OK, one weird issue I've found which is confounding but not fatal: the NodePort isn't working on v6. ` Capturing on 'eno1' 1 0.00... [16:17:25] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Show more useful information when mwscript-k8s fails to launch - https://phabricator.wikimedia.org/T369142#10196776 (10RLazarus) 05Open→03Resolved ` rzl@deploy1003:~$ mwscript-k8s --attach -- shell.php ⏳ Starting shell.php on Kubernetes as job mw-script.... [16:30:00] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10196898 (10taavi) Fwiw, the sync is now broken since https://wikitech.wikimedia.org/dumps/ is no longer served fr... [16:43:08] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10197002 (10Scott_French) Alright, at this point, the September 2024 switchover is officially done. There are still a couple of follow-ups to wrangle (see T3... [19:27:04] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10197621 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host mc-misc1001.eqiad.wmnet with OS bookworm [19:27:08] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10197622 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host mc-misc1002.eqiad.wmnet with OS bookworm [19:58:08] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10197709 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host mc-misc1002.eqiad.wmnet with OS bookworm completed: - mc-misc1002 (**PASS**)... [20:01:19] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10197724 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host mc-misc1001.eqiad.wmnet with OS bookworm completed: - mc-misc1001 (**PASS**)... [20:01:46] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10197726 (10Jclark-ctr) [20:02:36] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10197730 (10Jclark-ctr) a:05jijiki→03Jclark-ctr [20:04:05] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10197727 (10Jclark-ctr) 05Open→03Resolved @jijiki thanks for updating. they are finished now