[07:43:46] hello folks! [07:44:07] some deployments went through and the two new docker registry VMs on Bookworm seem to work fine [07:44:29] I am not going to propose a reimage the day of the switchover don't worry, but I'd start tomorrow if you are ok [07:45:25] elukey: stgm! [07:46:11] super :) [08:08:04] 06serviceops, 10MW-on-K8s, 10Scap: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10174389 (10jnuche) a:05jnuche→03akosiaris [08:09:19] 06serviceops, 10MW-on-K8s, 10Scap: Helm deployment timeouts during train presync - https://phabricator.wikimedia.org/T375477#10174383 (10jnuche) 05Open→03Resolved a:03jnuche Thank you @akosiaris! [08:52:45] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Debate whether we should grant the deploy ClusterRole permissions to create RoleBindings - https://phabricator.wikimedia.org/T375597#10174559 (10JMeybohm) Allowing the deploy user to create `Role` and `RoleBinding` objects within it's namespace would allow them... [11:20:20] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#10175060 (10akosiaris) [14:34:00] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10175891 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from eqiad to codf... [14:34:56] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10175892 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from... [14:42:19] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10175923 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw -... [14:46:26] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10175965 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to co... [14:53:10] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10175995 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to co... [14:58:55] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176020 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw... [14:59:27] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176021 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw... [15:02:40] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176038 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from eqiad to cod... [15:02:50] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176042 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from eqiad to co... [15:02:54] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176044 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw... [15:02:56] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176043 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from eqiad to co... [15:02:57] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176045 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw... [15:04:21] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176056 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad t... [15:08:09] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176063 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to c... [15:11:24] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176071 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw -... [15:12:55] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130#10176094 (10Trizek-WMF) Read-only phase is done. I removed the banner at 15:05 UTC. ----- Caught during the read-only phase: when y... [15:12:56] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130#10176098 (10Trizek-WMF) [15:22:07] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10176121 (10ops-monitoring-bot) swfrench@cumin1002 - Cookbook cookbooks.sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqi... [16:25:21] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645 (10elukey) 03NEW [16:26:07] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10176381 (10elukey) [16:30:25] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10176397 (10elukey) ` elukey@registry1003:~$ /usr/bin/docker-registry help garbage-collect `garbage-collect` deletes layers not referenced by a... [16:55:00] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10176545 (10elukey) ` elukey:~/go/pkg/mod/gitlab.com/gitlab-org/docker-distribution-pruner@v0.2.0$ EXPERIMENTAL=true ~/go/bin/docker-distributi...