[00:04:50] 06serviceops, 10Shellbox: Deploy Shellbox 4.1.1 server - https://phabricator.wikimedia.org/T381830#10392229 (10tstarling) [01:20:32] 06serviceops, 10MW-on-K8s, 06SRE-OnFire, 13Patch-For-Review, 10Sustainability (Incident Followup): mwscript-k8s creates too many resources - https://phabricator.wikimedia.org/T376795#10392357 (10RLazarus) Yes, naively this would be too many invocations at present. We could easily add the release name to... [01:27:03] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10392369 (10Scott_French) Thanks, @tstarling. Agreed that the risk of issues with Shellbox itself is likely quite low, and indeed the limited manual testing I've done in staging hasn't... [01:29:38] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381504#10392384 (10VRiley-WMF) [02:02:21] 06serviceops, 10Shellbox: Deploy Shellbox 4.1.1 server - https://phabricator.wikimedia.org/T381830#10392444 (10Scott_French) I'd propose that we jump straight to the latest image (`2024-12-07-073046`), which picks up two additional dependency updates in top of 4.1.1. Taken together, and filtering out test-onl... [02:29:57] 06serviceops, 10Shellbox: Deploy Shellbox 4.1.1 server - https://phabricator.wikimedia.org/T381830#10392461 (10tstarling) a:05tstarling→03Scott_French [08:32:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392626 (10ops-monitoring-bot) depool host kubernetes[1051-1054].eqiad.wmnet by jelto@cumin1002 with reason: Renaming nodes [08:34:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392633 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host kubernetes[1... [08:42:57] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392661 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1051 to wikikube-work... [08:49:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392687 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1052 to wikikube-work... [08:55:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392723 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1053 to wikikube-work... [09:00:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392750 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1054 to wikikube-work... [09:04:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392754 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1076.eq... [09:04:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392755 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1077.eq... [09:05:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392756 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1078.eq... [09:05:51] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392757 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1079.eq... [09:46:34] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392824 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1076.eqiad.wmnet with OS bookworm... [09:49:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392830 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1077.eqiad.wmnet with OS bookworm... [09:53:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392853 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1079.eqiad.wmnet with OS bookworm... [09:56:59] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392859 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1078.eqiad.wmnet with OS bookworm... [10:00:30] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392873 (10ops-monitoring-bot) pool host wikikube-worker[1076-1079].eqiad.wmnet by jelto@cumin1002 with reason: None [10:00:34] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10392874 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[1076-1079].eqiad.wmn... [10:01:04] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381504#10392875 (10Jelto) [10:54:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393020 (10ops-monitoring-bot) depool host kubernetes[1055-1058].eqiad.wmnet by jelto@cumin1002 with reason: Renaming nodes [10:56:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393023 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host kubernetes[1... [11:09:25] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393093 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1055 to wikikube-work... [11:13:59] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 10MediaWiki-Platform-Team (Radar), and 3 others: Create maintenance script to execute jobs provided in json format from standard input - https://phabricator.wikimedia.org/T369048#10393106 (10Krinkle) [11:15:35] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393121 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1056 to wikikube-work... [11:22:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393131 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1057 to wikikube-work... [11:29:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393144 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes1058 to wikikube-work... [11:32:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393155 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1080.eqiad.wmnet with OS book... [11:33:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393156 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1081.eqiad.wmnet with OS book... [11:33:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393160 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1082.eqiad.wmnet with OS book... [11:33:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393162 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1083.eqiad.wmnet with OS book... [11:35:57] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 10MediaWiki-Platform-Team (Radar), and 3 others: Create maintenance script to execute jobs provided in json format from standard input - https://phabricator.wikimedia.org/T369048#10393166 (10hnowlan) 05Open→03Resolved a:03hnowlan This script has bee... [12:11:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393377 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1082.eqiad.wmnet with OS bookworm... [12:16:12] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393381 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1080.eqiad.wmnet with OS bookworm... [12:19:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393388 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1083.eqiad.wmnet with OS bookworm... [12:53:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393484 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1081.eqiad.wmnet with OS bookworm... [12:55:51] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393491 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker1081.eqiad.wmnet with OS book... [14:03:16] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 07Kubernetes: Comm Error: backplane 0 when reimaging wikikube-worker1081 - https://phabricator.wikimedia.org/T381878 (10Jelto) 03NEW [14:04:29] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 07Kubernetes: Comm Error: backplane 0 when reimaging wikikube-worker1081 - https://phabricator.wikimedia.org/T381878#10393729 (10Jelto) The following commands have to be executed when the host is back (just noting it down so I don't forget it): ` c... [14:07:19] 06serviceops, 06Abstract Wikipedia team, 10function-evaluator: Have SRE provide a production-ready Rust image upstream - https://phabricator.wikimedia.org/T380807#10393744 (10Jdforrester-WMF) [14:16:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker1081.eqiad.wmnet with OS bookworm... [14:32:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393826 (10ops-monitoring-bot) pool host wikikube-worker[1080,1082-1083].eqiad.wmnet by jelto@cumin1002 with reason: None [14:32:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10393827 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[1080,1082-1083].eqia... [14:33:25] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381504#10393828 (10Jelto) [15:28:09] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381504#10394045 (10Jhancock.wm) @Jelto heads up, these are showing up in a netbox report. >Device is Active in Netbox but is missing from PuppetDB (should be ('d... [15:35:57] 06serviceops: Decommission kubernetes20[07-14].codfw.wmnet - https://phabricator.wikimedia.org/T379788#10394086 (10jasmine_) [15:52:08] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T381504#10394165 (10Jelto) >>! In T381504#10394045, @Jhancock.wm wrote: > @Jelto heads up, these are showing up in a netbox report. >>Device is Active in Netbox b... [17:09:33] 06serviceops: Package prometheus-mcrouter-exporter v0.4.0 - https://phabricator.wikimedia.org/T380212#10394448 (10jijiki) 05In progress→03Resolved a:03jijiki [17:41:24] 06serviceops, 10Shellbox: Deploy Shellbox 4.1.1 server - https://phabricator.wikimedia.org/T381830#10394645 (10Scott_French) Alright, the new image is live everywhere since ~ 17:13. The service appears healthy and I see no evidence of related errors / exceptions in logs on either the service- or mediawiki-side. [19:24:48] 06serviceops, 10Shellbox: Deploy Shellbox 4.1.1 server - https://phabricator.wikimedia.org/T381830#10394998 (10Scott_French) 05Open→03Resolved p:05Triage→03Medium Still no issues encountered after soaking for about 2h, so I'm going to call this resolved. [20:46:13] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 06SRE: wikikube-ctrl1002 and wikikube-ctrl1003: Switch network cable from port 2 to port 1 on the 10G NIC - https://phabricator.wikimedia.org/T379717#10395266 (10VRiley-WMF) Can we proceed with swapping these? [23:57:54] Hi there! Apologies if this is the wrong place to ask this. Our team is currently hiring an intern, and your team came up as one of the teams that recently had a successful internship round - would someone be willing to share with me whether you all provided a takehome assignment during the application process, and if so what it looked like?