[00:24:20] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10435830 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1257-1263].eqiad.wmnet} and (A:wiki... [07:32:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436228 (10ops-monitoring-bot) depool host wikikube-worker[2038-2039].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [07:34:02] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436229 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2038-2039].codfw.w... [07:37:10] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436232 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2039.codfw.wmnet with OS book... [07:49:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2038.codfw.wmnet with OS book... [08:27:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436452 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2039.codfw.wmnet with OS bookworm... [08:38:48] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436473 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2038.codfw.wmnet with OS bookworm... [08:48:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436495 (10ops-monitoring-bot) pool host wikikube-worker2039.codfw.wmnet by jelto@cumin1002 with reason: None [08:48:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436496 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2039.codfw.wmnet comp... [08:48:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436497 (10ops-monitoring-bot) pool host wikikube-worker2038.codfw.wmnet by jelto@cumin1002 with reason: None [08:49:01] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436498 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2038.codfw.wmnet comp... [08:53:57] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436504 (10ops-monitoring-bot) depool host wikikube-worker[2036-2037].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [08:55:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436519 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2036-2037].codfw.w... [08:57:07] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436520 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2036.codfw.wmnet with OS book... [08:57:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436522 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2037.codfw.wmnet with OS book... [09:27:53] hey folks! [09:28:16] https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DPuppetCertificateAboutToExpire shows some old termbox/mathoid/citoid cergen certs, I think we can drop them but lemme know [09:29:45] elukey: o/ - I'll re-open https://phabricator.wikimedia.org/T381474 to have a track record. Will take care of it, thanks [09:30:30] <3 [09:30:31] 06serviceops: Handle expiring puppet certificates - https://phabricator.wikimedia.org/T381474#10436593 (10JMeybohm) 05Resolvedβ†’03Open summary: Puppet CA certificate termbox.discovery.wmnet is about to expire summary: Puppet CA certificate mathoid.discovery.wmnet is about to expire summary: Puppet CA certific... [09:40:25] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436602 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2036.codfw.wmnet with OS bookworm... [09:46:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436615 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2037.codfw.wmnet with OS bookworm... [10:01:25] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436655 (10ops-monitoring-bot) pool host wikikube-worker2037.codfw.wmnet by jelto@cumin1002 with reason: None [10:01:26] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436656 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2037.codfw.wmnet comp... [10:01:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436657 (10ops-monitoring-bot) pool host wikikube-worker2036.codfw.wmnet by jelto@cumin1002 with reason: None [10:01:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436658 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2036.codfw.wmnet comp... [10:02:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436661 (10ops-monitoring-bot) depool host wikikube-worker[2034-2035].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [10:04:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436664 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2034-2035].codfw.w... [10:05:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436665 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2035.codfw.wmnet with OS book... [10:05:51] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436666 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2034.codfw.wmnet with OS book... [10:46:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436755 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2034.codfw.wmnet with OS bookworm... [10:51:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436760 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2035.codfw.wmnet with OS bookworm... [10:52:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436769 (10ops-monitoring-bot) pool host wikikube-worker2035.codfw.wmnet by jelto@cumin1002 with reason: None [10:52:12] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436770 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2035.codfw.wmnet comp... [10:52:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436771 (10ops-monitoring-bot) pool host wikikube-worker2034.codfw.wmnet by jelto@cumin1002 with reason: None [10:52:23] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436772 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2034.codfw.wmnet comp... [10:53:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436777 (10ops-monitoring-bot) depool host wikikube-worker[2032-2033].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [10:57:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436779 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2032-2033].codfw.w... [11:01:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436793 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2033.codfw.wmnet with OS book... [11:01:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436794 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2032.codfw.wmnet with OS book... [11:41:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436861 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2033.codfw.wmnet with OS bookworm... [11:52:59] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436895 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2032.codfw.wmnet with OS bookworm... [11:58:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436905 (10ops-monitoring-bot) pool host wikikube-worker2033.codfw.wmnet by jelto@cumin1002 with reason: None [11:58:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436906 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2033.codfw.wmnet comp... [11:58:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436909 (10ops-monitoring-bot) pool host wikikube-worker2032.codfw.wmnet by jelto@cumin1002 with reason: None [11:58:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436910 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2032.codfw.wmnet comp... [11:59:34] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436914 (10ops-monitoring-bot) depool host wikikube-worker[2030-2031].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [12:00:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436920 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2030-2031].codfw.w... [12:01:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436921 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2031.codfw.wmnet with OS book... [12:01:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10436922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2030.codfw.wmnet with OS book... [12:44:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437023 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2031.codfw.wmnet with OS bookworm... [12:46:43] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10437025 (10ops-monitoring-bot) Roll-reimage of nodes in wikikube-eqiad cluster started by jayme: * wikikube-worker[1253-1255].eqiad.wmnet [12:47:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437028 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2030.codfw.wmnet with OS bookworm... [12:47:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10437029 (10ops-monitoring-bot) Roll-reimage of nodes in wikikube-eqiad cluster started by jayme: * wikikube-worker[1264-1266].eqiad.wmnet [13:15:52] swfrench-wmf: Hiii, I have a question you might know, if not, please point me in the right direction. How can we via dbctl, fully depool a whole section [13:51:07] Amir1: I don't think we ever meant to allow that πŸ˜… [13:51:47] cdanis: I need it for pc sections [13:52:00] some context: https://phabricator.wikimedia.org/T373037 [13:53:08] I sorta think if we force depool or set weight of all the hosts in that section to zero, it should work but obviously can't test it in production :D [13:55:19] (when I was setting up pc5, everything was correct but weight of the hosts were zero, because of that pc5 wasn't showing up at all) [14:00:35] hmm [14:00:46] we could use a dbctl testbed for sure [14:06:34] ooh, where is it? This is the first time I'm hearing abot it [14:08:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437224 (10ops-monitoring-bot) pool host wikikube-worker2030.codfw.wmnet by jelto@cumin1002 with reason: None [14:08:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437225 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2030.codfw.wmnet comp... [14:08:20] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437226 (10ops-monitoring-bot) pool host wikikube-worker2031.codfw.wmnet by jelto@cumin1002 with reason: None [14:08:23] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437227 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2031.codfw.wmnet comp... [14:08:58] cdanis: is there a doc somewhere? [14:09:07] oh it doesn't exist [14:09:15] I'm saying it *would* be useful πŸ˜… [14:09:35] it's not too hard to set one up yourself but maybe I'll look at automating that a little [14:09:47] take some of joe's best practices for local dev environments [14:11:01] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437234 (10ops-monitoring-bot) depool host wikikube-worker[2028-2029].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [14:12:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437239 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2028-2029].codfw.w... [14:13:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437240 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2028.codfw.wmnet with OS book... [14:13:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437241 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2029.codfw.wmnet with OS book... [14:13:30] ah, okay [14:33:24] 06serviceops, 10Citoid, 10VisualEditor, 10VisualEditor-MediaWiki-References, and 2 others: Register Citoid as a "friendly bot" (or alternatively verified bot) with Cloudflare - https://phabricator.wikimedia.org/T370118#10437328 (10VPuffetMichel) I verified and citoid is now a verified bot. https://radar.c... [14:40:58] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), 07User-notice: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10437347 (10akosiaris) I think we can inde... [14:55:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437382 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2028.codfw.wmnet with OS bookworm... [14:58:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437385 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2029.codfw.wmnet with OS bookworm... [15:07:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437409 (10ops-monitoring-bot) pool host wikikube-worker2029.codfw.wmnet by jelto@cumin1002 with reason: None [15:07:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437410 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2029.codfw.wmnet comp... [15:07:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437411 (10ops-monitoring-bot) pool host wikikube-worker2028.codfw.wmnet by jelto@cumin1002 with reason: None [15:07:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437412 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2028.codfw.wmnet comp... [15:08:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437415 (10ops-monitoring-bot) depool host wikikube-worker[2026-2027].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [15:10:07] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437416 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2026-2027].codfw.w... [15:11:32] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437418 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2027.codfw.wmnet with OS book... [15:11:35] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437419 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2026.codfw.wmnet with OS book... [15:22:23] 06serviceops, 10Citoid, 10VisualEditor, 10VisualEditor-MediaWiki-References, and 2 others: Register Citoid as a "friendly bot" (or alternatively verified bot) with Cloudflare - https://phabricator.wikimedia.org/T370118#10437458 (10akosiaris) 05Openβ†’03Resolved Wooohooo. Thanks for taking over and ha... [15:27:41] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), 07User-notice: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10437475 (10HCoplin-WMF) Excellent! I'll l... [15:31:20] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10437489 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1253-1255].eqiad.wmnet} and (A:wiki... [15:39:02] 10:38:29 (I’m still not deploying btw, trying to figure out first how I can verify the new shellbox version) [15:40:33] Amir1: yeah, similar to cdanis' response, I don't think that's possible (or intended to be possible) at the moment. if it's a feature that you anticipate having some use for - e.g., as a special-case for parsercache - it could be made possible :) [15:40:33] ... but that opens some interesting cans of worms about what a "valid" config might be [15:51:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10437532 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1264-1266].eqiad.wmnet} and (A:wiki... [15:55:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437549 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2027.codfw.wmnet with OS bookworm... [15:55:38] 06serviceops, 06Abstract Wikipedia team, 10function-evaluator, 13Patch-For-Review: Have SRE provide a production-ready Rust image upstream - https://phabricator.wikimedia.org/T380807#10437550 (10akosiaris) > As discussed with your team which led to this task, this is because your team enforces the helm res... [15:58:52] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437564 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2026.codfw.wmnet with OS bookworm... [15:59:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437566 (10ops-monitoring-bot) pool host wikikube-worker2027.codfw.wmnet by jelto@cumin1002 with reason: None [15:59:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437567 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2027.codfw.wmnet comp... [15:59:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437568 (10ops-monitoring-bot) pool host wikikube-worker2026.codfw.wmnet by jelto@cumin1002 with reason: None [15:59:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10437569 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2026.codfw.wmnet comp... [15:59:49] Amir1: swfrench-wmf: definitely worth opening a task [16:06:12] cdanis: T383137 [16:06:58] funny enough, the "very defensive" way `wmgPCServers` is populated means that if one were to commit a config with no leader for the desired section in externalLoads, this would totally work (i.e., the section would become invisible to SqlBagOStuff) [16:07:30] Amir1: thanks! [16:07:50] that's enough for me too :D [16:08:18] I want to test it but not in production :P do you have a setup handy swfrench-wmf ? [16:09:35] alas, I do not =/ (the testing I did when adding the "parsercache as external store" functionality only tested the etcd write path locally - not the mediawiki config side of things, which is the more interesting part) [16:10:31] so, I should mention, the tool won't let you do what I described above, as all sections must have exactly one leader, per the validation checks applied when committing the config [16:10:54] what if that leader was zero weight? :D [16:11:19] alas, it doesn't care what the weight is =/ [16:11:45] :( [16:11:55] (similar to core sections, where the leader can have zero weight, and that's totally fine / ignored) [16:12:02] 06serviceops, 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10437672 (10BTullis) [16:13:36] in any case, I'll give this some thought today [16:18:46] swfrench-wmf: something that I've been thinking [16:19:10] we have been calling them 'flavors' so far but maybe it's better to think of them as, like, mixins, or aspects [16:19:20] or feature flags [16:19:38] pc sections get the EmptyIsAllowed trait, or something [16:23:47] having a use-case-specific set of validation guardrails would be pretty useful for something like this, yeah :) [16:24:08] i.e., "this is parsercache, use that validation profile" [16:51:20] 06serviceops: Handle expiring puppet certificates - https://phabricator.wikimedia.org/T381474#10437821 (10JMeybohm) 05Openβ†’03Resolved [17:11:37] swfrench-wmf: wow, that etcd.php code is defensive [17:12:08] yes :) [17:13:26] Amir1: well, I had a very picky reviewer :P [17:13:54] :D [17:15:54] made the patch https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1108794 [17:18:15] I'll give a bit for Scott and then deploy it tomorrow [17:18:36] nice! that certainly looks like it should do the job, yeah - I'll take a look today after wrapping up what I'm wrangling at the moment [17:18:37] Will do a test depool of pc5 as well. If all works well, we start to get rid of spare [17:18:49] *spares [17:18:55] Will need to update the docs too [17:50:57] 06serviceops: Improve helmfile-only update workflow for mw-videoscaler - https://phabricator.wikimedia.org/T382630#10438068 (10Scott_French) Alright, the interim "release generation" approach is now live and appears to work as expected. Further, the now far-shorter job names are a lot less concerning w.r.t. uniq... [18:04:14] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03), 13Patch-For-Review: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10438160 (10Scott_French) The rsyslog container is now up and running, and the first... [20:04:21] https://trace.wikimedia.org/trace/f5a733d4bd4c217dab217ef9852f7340 πŸŽ‰ [20:17:12] cdanis: very nice! :) [20:40:18] 21:38:45 <β€’icinga-wm> PROBLEM - Host doc1003 is DOWN: PING CRITICAL - Packet loss = 100% [20:40:18] :/ [20:40:25] sorry wrong channel