[03:11:27] 06serviceops, 10MW-on-K8s, 10Shellbox, 10SRE-swift-storage, and 3 others: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322#10431257 (10tstarling) I confirmed that this is working on testwiki. [05:06:30] 06serviceops, 10MW-on-K8s, 10Shellbox, 10SRE-swift-storage, and 2 others: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322#10431315 (10tstarling) 05Open→03Resolved I did a new benchmark with a method following T292322#10402444. The time taken from the job start to ffmpeg... [07:55:52] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431484 (10ops-monitoring-bot) depool host wikikube-worker[2053-2054].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to... [07:59:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431485 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-wor... [08:00:43] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431486 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2053.co... [08:00:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431487 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2054.co... [08:36:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10431592 (10ops-monitoring-bot) Roll-reimage of nodes in wikikube-eqiad cluster started by jayme: * wikikube-worker[1240-1244].eqiad.wmnet [08:37:53] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10431595 (10ops-monitoring-bot) Roll-reimage of nodes in wikikube-eqiad cluster started by jayme: * wikikube-worker[1245-1249].eqiad.wmnet [08:42:05] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431600 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2054.codfw.... [08:42:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431613 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2053.codfw.... [08:43:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431615 (10ops-monitoring-bot) pool host wikikube-worker[2053-2054].codfw.wmnet by jelto@cumin1002 with reason: None [08:44:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431619 (10ops-monitoring-bot) pool host wikikube-worker[2053-2054].codfw.wmnet by jelto@cumin1002 with reason: None [08:46:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431623 (10ops-monitoring-bot) pool host wikikube-worker2054.codfw.wmnet by jelto@cumin1002 with reason: None [08:46:17] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431624 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [08:46:25] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431625 (10ops-monitoring-bot) pool host wikikube-worker2053.codfw.wmnet by jelto@cumin1002 with reason: None [08:46:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431626 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [08:56:10] 06serviceops, 06Abstract Wikipedia team: wikifunction httpbb tests fail because of title case issue - https://phabricator.wikimedia.org/T383032 (10Clement_Goubert) 03NEW [09:04:24] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431664 (10ops-monitoring-bot) depool host wikikube-worker[2051-2052].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to... [09:05:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431668 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-wor... [09:06:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431669 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2051.co... [09:06:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431670 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2052.co... [09:43:29] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Port videoscaling to kubernetes - https://phabricator.wikimedia.org/T355292#10431750 (10Clement_Goubert) @tstarling @TheDJ This has been flagged in {T382517} and is due to the `mw-videoscaler` deployment missing the `rsyslog` si... [09:47:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431761 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2051.codfw.... [09:52:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431770 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2052.codfw.... [09:52:23] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431771 (10ops-monitoring-bot) pool host wikikube-worker2051.codfw.wmnet by jelto@cumin1002 with reason: None [09:52:25] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431772 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [09:52:35] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431773 (10ops-monitoring-bot) pool host wikikube-worker2052.codfw.wmnet by jelto@cumin1002 with reason: None [09:52:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431774 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [09:53:43] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431780 (10ops-monitoring-bot) depool host wikikube-worker[2049-2050].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to... [09:54:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431781 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-wor... [09:55:26] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431787 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2049.co... [09:55:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431788 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2050.co... [10:36:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431929 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2050.codfw.wmnet with OS bookworm... [10:39:20] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431935 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2049.codfw.wmnet with OS bookworm... [10:49:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431989 (10ops-monitoring-bot) pool host wikikube-worker2049.codfw.wmnet by jelto@cumin1002 with reason: None [10:49:10] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431990 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2049.codfw.wmnet comp... [10:49:34] dear serviceops folks. How angry you would be if I run a maint script in eqiad? From 1 to 10. Context: https://phabricator.wikimedia.org/T382948 [10:49:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431996 (10ops-monitoring-bot) pool host wikikube-worker2050.codfw.wmnet by jelto@cumin1002 with reason: None [10:49:41] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10431997 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2050.codfw.wmnet comp... [10:53:23] Hmm good question [10:53:38] probably not angry but disappointed [10:53:58] all joking aside, and short answer is idk [10:56:23] Amir1: Mostly because afaiu eqiad being RO is mostly for the databases' sake to avoid replication issues/split brain (feel free to correct me) [10:57:06] I think I need an adult to answer this, akosiaris you around? [11:06:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432030 (10ops-monitoring-bot) depool host wikikube-worker[2046,2048].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [11:08:02] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432038 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2046,2048].codfw.w... [11:08:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432044 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2048.codfw.wmnet with OS book... [11:08:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432045 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2046.codfw.wmnet with OS book... [11:10:16] ah dang, ooo today [11:13:30] from db point of view, it shouldn't be an issue, we have code paths that do writes on GET from secondary dc. They just connect to the remote master (and are slow) but they are fine. [11:52:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432137 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2048.codfw.wmnet with OS bookworm... [11:55:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432138 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2046.codfw.wmnet with OS bookworm... [12:30:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10432184 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1245-1249].eqiad.wmnet} and (A:wiki... [12:51:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432217 (10ops-monitoring-bot) pool host wikikube-worker2048.codfw.wmnet by jelto@cumin1002 with reason: None [12:51:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432218 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2048.codfw.wmnet comp... [12:51:43] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432219 (10ops-monitoring-bot) pool host wikikube-worker2046.codfw.wmnet by jelto@cumin1002 with reason: None [12:51:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432220 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2046.codfw.wmnet comp... [12:52:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432221 (10ops-monitoring-bot) depool host wikikube-worker[2044-2045].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [12:53:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432227 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2044-2045].codfw.w... [12:54:51] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432230 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2045.codfw.wmnet with OS book... [12:54:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432231 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2044.codfw.wmnet with OS book... [13:25:51] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 2 others: Comm Error: backplane 0 when reimaging wikikube-worker1243 - https://phabricator.wikimedia.org/T383051 (10JMeybohm) 03NEW [13:29:01] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 2 others: Comm Error: backplane 0 when reimaging wikikube-worker1243 - https://phabricator.wikimedia.org/T383051#10432351 (10JMeybohm) The following commands have to be executed when the host is back (just noting it down so I don't forget i... [13:34:46] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432363 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2044.codfw.wmnet with OS bookworm... [13:35:48] 06serviceops, 06Abstract Wikipedia team, 10function-evaluator, 13Patch-For-Review: Have SRE provide a production-ready Rust image upstream - https://phabricator.wikimedia.org/T380807#10432364 (10Jdforrester-WMF) >>! In T380807#10418781, @akosiaris wrote: >>>! In T380807#10413489, @Jdforrester-WMF wrote: >>... [13:38:02] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: hw troubleshooting: "Comm Error: backplane 0" for reimaging wikikube-worker1243 - https://phabricator.wikimedia.org/T383051#10432376 (10JMeybohm) [13:39:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432383 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2045.codfw.wmnet with OS bookworm... [13:39:46] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: hw troubleshooting: "Comm Error: backplane 0" for reimaging wikikube-worker1243 - https://phabricator.wikimedia.org/T383051#10432382 (10JMeybohm) a:03Jclark-ctr [13:40:15] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1243 - https://phabricator.wikimedia.org/T383051#10432384 (10JMeybohm) [13:40:25] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432385 (10ops-monitoring-bot) pool host wikikube-worker2045.codfw.wmnet by jelto@cumin1002 with reason: None [13:40:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432386 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2045.codfw.wmnet comp... [13:40:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432388 (10ops-monitoring-bot) pool host wikikube-worker2044.codfw.wmnet by jelto@cumin1002 with reason: None [13:40:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432389 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2044.codfw.wmnet comp... [13:41:04] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1057 - https://phabricator.wikimedia.org/T381676#10432390 (10JMeybohm) a:03Jclark-ctr [13:42:08] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1243.eqiad.wmnet - https://phabricator.wikimedia.org/T383051#10432395 (10JMeybohm) [13:42:46] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1057.eqiad.wmnet - https://phabricator.wikimedia.org/T381676#10432397 (10JMeybohm) [13:43:32] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1069.eqiad.wmnet - https://phabricator.wikimedia.org/T381770#10432399 (10JMeybohm) a:03Jclark-ctr [13:44:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1073.eqiad.wmnet - https://phabricator.wikimedia.org/T381789#10432405 (10JMeybohm) [13:44:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1073.eqiad.wmnet - https://phabricator.wikimedia.org/T381789#10432407 (10JMeybohm) a:03Jclark-ctr [13:45:37] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T381878#10432410 (10JMeybohm) [13:46:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432415 (10ops-monitoring-bot) depool host wikikube-worker[2042-2043].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [13:47:25] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432416 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2042-2043].codfw.w... [13:47:26] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10432417 (10ops-monitoring-bot) Roll-reimage of nodes in wikikube-eqiad cluster started by jayme: * wikikube-worker[1250-1252].eqiad.wmnet [13:47:39] 06serviceops, 10ChangeProp, 10EventStreams, 10Recommendation-API, and 2 others: Migrate node-based services in production to node12 - https://phabricator.wikimedia.org/T290750#10432418 (10Jdforrester-WMF) [13:49:30] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432419 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2043.codfw.wmnet with OS book... [13:49:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432420 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2042.codfw.wmnet with OS book... [14:30:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432627 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2043.codfw.wmnet with OS bookworm... [14:33:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432643 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2042.codfw.wmnet with OS bookworm... [14:34:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432644 (10ops-monitoring-bot) pool host wikikube-worker2043.codfw.wmnet by jelto@cumin1002 with reason: None [14:34:35] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432646 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2043.codfw.wmnet comp... [14:34:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432657 (10ops-monitoring-bot) pool host wikikube-worker2042.codfw.wmnet by jelto@cumin1002 with reason: None [14:34:41] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432658 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2042.codfw.wmnet comp... [14:35:26] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432659 (10ops-monitoring-bot) depool host wikikube-worker[2040-2041].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [14:39:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432673 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2040-2041].codfw.w... [14:40:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432676 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2041.codfw.wmnet with OS book... [14:40:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432677 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2040.codfw.wmnet with OS book... [15:24:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432836 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2041.codfw.wmnet with OS bookworm... [15:29:46] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432861 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2040.codfw.wmnet with OS bookworm... [15:31:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432868 (10ops-monitoring-bot) pool host wikikube-worker2040.codfw.wmnet by jelto@cumin1002 with reason: None [15:31:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432869 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2040.codfw.wmnet comp... [15:31:30] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432870 (10ops-monitoring-bot) pool host wikikube-worker2041.codfw.wmnet by jelto@cumin1002 with reason: None [15:31:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10432871 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2041.codfw.wmnet comp... [15:46:01] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03), 13Patch-For-Review: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10432893 (10Scott_French) [15:46:02] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Port videoscaling to kubernetes - https://phabricator.wikimedia.org/T355292#10432894 (10Scott_French) [16:57:13] Is there a task anywhere in Phabricator for upgrading the base image for the Shellbox containers to bullseye or bookworm? Context is upgrading Pygments for Extension:SyntaxHighlight. We need Python 3.8+ for the T364249 and now T383018 updates. [16:59:59] bd808: I think swfrench-wmf might be looking at that as an aspect of https://phabricator.wikimedia.org/T377038 but I'm not positive [17:00:21] (and we're all heading into a meeting so responses might be slowish) [17:00:56] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), 07User-notice: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10433233 (10HCoplin-WMF) Hello! Since we'r... [17:02:17] bd808: yes, I'll be bumping them to bullseye (so 3.9) as part of the 8.1 update, and the syntaxhighlight variant is actually the first one I'll be doing (ideally starting this or next week) [17:02:21] the php 8.1 version of the shellbox image is bullseye [17:02:31] heh 4 seconds too slow [17:03:24] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10433243 (10bd808) [17:05:13] swfrench-wmf: excellent. I will block on that task then and look forward to being able to resumen updates of SyntaxHighlight in the not too distant future. :) [17:05:23] *resume [17:15:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10433297 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1240-1244].eqiad.wmnet} and (A:wiki... [17:16:02] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10433298 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1250-1252].eqiad.wmnet} and (A:wiki... [17:17:02] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10433303 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker1244.eqiad.wmnet with OS book... [17:59:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10433534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker1244.eqiad.wmnet with OS bookworm... [18:46:22] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485#10433708 (10Jdlrobson) [19:09:05] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10433783 (10ops-monitoring-bot) Roll-reimage of nodes in wikikube-eqiad cluster started by kamila: * wikikube-worker[1257-1261].eqiad.wmnet [19:09:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10433784 (10ops-monitoring-bot) Roll-reimage of nodes in wikikube-eqiad cluster started by kamila: * wikikube-worker[1257-1263].eqiad.wmnet [20:55:49] 06serviceops, 10AQS2.0, 06Data-Engineering: Investigate effort in routing device analytics via istio - https://phabricator.wikimedia.org/T358324#10434247 (10VirginiaPoundstone) [21:14:14] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform: Make eventstreams-internal available to WMF staff without an ssh tunnel - https://phabricator.wikimedia.org/T348763#10434465 (10VirginiaPoundstone) [21:15:01] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 10 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#10434466 (10VirginiaPoundstone) [21:18:08] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Proton, and 3 others: WikiKube: Rename the last few "production" named helm releases to use "main" instead - https://phabricator.wikimedia.org/T377805#10434525 (10VirginiaPoundstone) [21:33:47] 06serviceops, 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10434814 (10VirginiaPoundstone) [21:34:56] 06serviceops, 06Data-Engineering-Icebox, 06SRE, 06Trust-and-Safety: Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset - https://phabricator.wikimedia.org/T303464#10434838 (10VirginiaPoundstone) [21:55:03] 06serviceops, 13Patch-For-Review: Improve helmfile-only update workflow for mw-videoscaler - https://phabricator.wikimedia.org/T382630#10435528 (10Scott_French) [22:45:44] 06serviceops, 13Patch-For-Review: Improve helmfile-only update workflow for mw-videoscaler - https://phabricator.wikimedia.org/T382630#10435717 (10Scott_French) After a bit more thought and experiments, I think an "optional release generation" approach like that in https://gerrit.wikimedia.org/r/1105821 is pro...