[01:02:56] 06serviceops, 10Observability-Metrics, 07Wikimedia-Performance-recommendation: Enable mediawiki appserver metrics for jobrunner hosts - https://phabricator.wikimedia.org/T293943#10446664 (10colewhite) Are we capturing this data now that most everything is in k8s? [08:25:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447282 (10ops-monitoring-bot) pool host wikikube-worker1057.eqiad.wmnet by jelto@cumin1002 with reason: None [08:25:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447283 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker1057.eqiad.wmnet comp... [08:30:02] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1057.eqiad.wmnet - https://phabricator.wikimedia.org/T381676#10447286 (10Jelto) Thanks @Jclark-ctr for the quick help and running the reimage one more time. The host looks good to me now. I exe... [08:32:02] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447288 (10ops-monitoring-bot) pool host wikikube-worker1069.eqiad.wmnet by jelto@cumin1002 with reason: None [08:32:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447289 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker1069.eqiad.wmnet comp... [08:32:11] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1069.eqiad.wmnet - https://phabricator.wikimedia.org/T381770#10447291 (10Jelto) Thanks @Jclark-ctr for the quick help and running the reimage one more time. The host lo... [08:34:35] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447293 (10ops-monitoring-bot) pool host wikikube-worker1073.eqiad.wmnet by jelto@cumin1002 with reason: None [08:34:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447294 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker1073.eqiad.wmnet comp... [08:34:49] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1073.eqiad.wmnet - https://phabricator.wikimedia.org/T381789#10447296 (10Jelto) Thanks @Jclark-ctr for the quick help and running the reimage one more time. The host lo... [08:37:05] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447298 (10ops-monitoring-bot) pool host wikikube-worker1081.eqiad.wmnet by jelto@cumin1002 with reason: None [08:37:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447299 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker1081.eqiad.wmnet comp... [08:38:50] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T381878#10447301 (10Jelto) Thanks @Jclark-ctr for the quick help and running the reimage one more time. The host lo... [08:39:53] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447305 (10ops-monitoring-bot) pool host wikikube-worker1243.eqiad.wmnet by jelto@cumin1002 with reason: None [08:39:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10447306 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker1243.eqiad.wmnet comp... [08:39:56] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-eqiad, and 3 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1243.eqiad.wmnet - https://phabricator.wikimedia.org/T383051#10447308 (10Jelto) Thanks @Jclark-ctr for the quick help and running the reimage one more time. The... [09:21:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447351 (10ops-monitoring-bot) pool host wikikube-worker2022.codfw.wmnet by jelto@cumin1002 with reason: None [09:21:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447352 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [09:26:00] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447361 (10ops-monitoring-bot) depool host kubernetes[2049-2052].codfw.wmnet by jelto@cumin1002 with reason: Renaming nodes [09:28:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447362 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host kubernetes[2... [09:38:59] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447379 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes2049 to wikikube-work... [09:45:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447382 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes2050 to wikikube-work... [09:51:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447389 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes2051 to wikikube-work... [09:57:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447401 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes2052 to wikikube-work... [10:00:48] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447404 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2195.co... [10:00:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447405 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2196.co... [10:30:50] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10447455 (10ops-monitoring-bot) pool host wikikube-worker[1093-1095].eqiad.wmnet by kamila@cumin1002 with reason: None [10:30:54] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10447456 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1093-1095].eqia... [10:32:33] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T383213#10447459 (10kamila) [10:49:12] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447662 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2196.codfw.wmnet with OS bookworm... [10:52:00] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447663 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2195.codfw.wmnet with OS bookworm... [10:54:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447671 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2198.codfw.wmnet with OS book... [10:54:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447670 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2197.codfw.wmnet with OS book... [11:42:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447789 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2198.codfw.wmnet with OS bookworm... [11:46:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447790 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2197.codfw.wmnet with OS bookworm... [11:54:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447812 (10ops-monitoring-bot) pool host wikikube-worker[2195-2198].codfw.wmnet by jelto@cumin1002 with reason: None [11:54:17] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447813 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[2195-2198].codfw.wmn... [11:54:59] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T383341#10447829 (10Jelto) [12:58:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447959 (10ops-monitoring-bot) depool host kubernetes[2045-2048].codfw.wmnet by jelto@cumin1002 with reason: Renaming nodes [13:01:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447965 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host kubernetes[2... [13:09:52] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447973 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes2045 to wikikube-work... [13:17:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10447999 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes2046 to wikikube-work... [13:24:05] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448014 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes2047 to wikikube-work... [13:30:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448028 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from kubernetes2048 to wikikube-worker2202 completed: - ku... [13:35:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448037 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2199.codfw.wmnet with OS book... [13:35:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448038 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2200.codfw.wmnet with OS book... [14:05:17] 06serviceops, 10decommission-hardware: decommission mw135[8-9], mw136[4-6], mw137[2-3], mw140[0-4], mw1406, mw14[11-13] - https://phabricator.wikimedia.org/T383227#10448142 (10kamila) [14:23:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448202 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2199.codfw.wmnet with OS bookworm... [14:24:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448203 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2201.codfw.wmnet with OS book... [14:26:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448229 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2200.codfw.wmnet with OS bookworm... [14:26:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448240 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2202.codfw.wmnet with OS book... [14:28:15] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 3 others: Generate a dumps-enabled mediawiki image - https://phabricator.wikimedia.org/T381473#10448251 (10Gehel) [14:35:18] 06serviceops, 06Traffic: Investigate why pools.json does not match https://config-master.wikimedia.org/pybal/${datacenter}/${service} T363702 - https://phabricator.wikimedia.org/T364037#10448344 (10Gehel) [15:18:56] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 4 others: Generate a dumps-enabled mediawiki image - https://phabricator.wikimedia.org/T381473#10448513 (10Gehel) [15:25:43] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448560 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2201.codfw.wmnet with OS bookworm... [15:28:34] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Remove the kubelet readOnlyPort - https://phabricator.wikimedia.org/T383413 (10JMeybohm) 03NEW [15:32:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448642 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2202.codfw.wmnet with OS bookworm... [15:36:34] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448686 (10ops-monitoring-bot) pool host wikikube-worker[2199-2202].codfw.wmnet by jelto@cumin1002 with reason: None [15:36:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10448700 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[2199-2202].codfw.wmn... [15:38:13] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T383341#10448708 (10Jelto) [15:47:57] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Remove the kubelet readOnlyPort - https://phabricator.wikimedia.org/T383413#10448790 (10JMeybohm) a:03JMeybohm [15:49:32] 06serviceops, 06Traffic: Investigate why pools.json does not match https://config-master.wikimedia.org/pybal/${datacenter}/${service} T363702 - https://phabricator.wikimedia.org/T364037#10448808 (10Gehel) [16:18:51] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Prod-Kubernetes, 10Wikidata, 10Wikidata-Query-Service: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes - https://phabricator.wikimedia.org/T293063#10448952 (10Gehel) [16:58:35] 06serviceops, 06Data-Engineering: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#10449268 (10Gehel) [17:13:58] 06serviceops, 06Growth-Team, 10Notifications, 10wikitech.wikimedia.org, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449347 (10Ladsgroup) I want to double check in a mw-api-ext pod but curl doesn't exist there ` root@deploy2... [17:30:14] 06serviceops, 06Growth-Team, 10Notifications, 10wikitech.wikimedia.org, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449378 (10CDanis) https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Exec_into_a_pod_and_run_co... [17:38:10] 06serviceops, 06Growth-Team, 10Notifications, 10wikitech.wikimedia.org, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449389 (10Ladsgroup) Thanks! Now I confirm that it's a firewall issue on the whole pod: ` root@wikikube-wo... [17:39:48] 06serviceops, 06Growth-Team, 10Notifications, 10wikitech.wikimedia.org, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449390 (10taavi) AIUI MediaWiki should be sending that request directly to itself instead of going through... [17:46:35] 06serviceops, 06Growth-Team, 10Notifications, 10wikitech.wikimedia.org, and 2 others: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449423 (10Ladsgroup) This should fix it. [19:03:33] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10449562 (10Scott_French) Alright, regrouping post-holiday and pulling together more thoughts on how to migrate traffic for shellbox-syntaxhighlight: First, beyond an initial pilot wit... [19:05:30] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10449577 (10Scott_French) 05Open→03In progress p:05Triage→03Medium