[01:14:28] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Steady-state sizing of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T376519#10439450 (10Scott_French) Following up - back in October, we held off on reverting to pre-switchover sizing (https://gerrit.wikimedia.org/r/1078481) in advance o... [01:50:51] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03): PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10439555 (10Scott_French) For better or worse, it has been a quiet day in terms of errors emitted by mw-vi... [04:13:17] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03), 13Patch-For-Review: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10439646 (10Scott_French) Later this evening, I realized that I was too focused on v... [07:44:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10439799 (10ops-monitoring-bot) depool host wikikube-worker[2024-2025].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [07:46:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10439800 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2024-2025].codfw.w... [07:47:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10439802 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2024.codfw.wmnet with OS book... [07:47:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10439803 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2025.codfw.wmnet with OS book... [08:19:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10439831 (10ops-monitoring-bot) Roll-reimage of nodes in wikikube-eqiad cluster started by jayme: * wikikube-worker[1267-1269].eqiad.wmnet [08:26:15] hey folks o/ [08:26:29] I see that there is a pending cert request for kubernetes1021 on puppetserver1001 [08:26:32] https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DPuppetPendingCertificateRequest [08:26:49] should we clean it up? It seems cruft from past work [08:34:20] elukey: yeah, that can be dropped I suppose. kubernetes1021 does not exist anymore [08:35:08] okok! [08:35:10] 06serviceops, 06Infrastructure-Foundations, 07Security: Upgrade K8s docker images running in Wikimedia production on Buster to either Bullseye or Bookworm - https://phabricator.wikimedia.org/T368366#10439920 (10elukey) [08:36:18] thanks! [09:08:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10439988 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2025.codfw.wmnet with OS bookworm... [09:10:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10439989 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2025.codfw.wmnet with OS book... [09:12:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10439993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2024.codfw.wmnet with OS bookworm... [09:13:12] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10439994 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2024.codfw.wmnet with OS book... [09:39:39] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Steady-state sizing of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T376519#10440039 (10Clement_Goubert) SGTM, thanks for doing the maths [09:51:08] 06serviceops, 06Abstract Wikipedia team: wikifunction httpbb tests fail because of title case issue - https://phabricator.wikimedia.org/T383032#10440051 (10Clement_Goubert) p:05Triage→03High [09:52:02] 06serviceops, 06Abstract Wikipedia team: wikifunction httpbb tests fail because of title case issue - https://phabricator.wikimedia.org/T383032#10440053 (10Clement_Goubert) Re-prioritizing this to High, as the alert is actually critical, and would mask other wikifunctions `httpbb` alerts if they were to happen. [09:55:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440055 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2024.codfw.wmnet with OS bookworm... [09:59:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440057 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2025.codfw.wmnet with OS bookworm... [10:02:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440061 (10ops-monitoring-bot) pool host wikikube-worker2025.codfw.wmnet by jelto@cumin1002 with reason: None [10:02:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440062 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2025.codfw.wmnet comp... [10:02:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440063 (10ops-monitoring-bot) pool host wikikube-worker2024.codfw.wmnet by jelto@cumin1002 with reason: None [10:02:52] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440064 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2024.codfw.wmnet comp... [10:03:52] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Steady-state sizing of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T376519#10440068 (10akosiaris) SGTM as well [10:14:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440087 (10ops-monitoring-bot) depool host wikikube-worker[2022-2023].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [10:15:51] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440090 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2022-2023].codfw.w... [10:19:30] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440094 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS book... [10:19:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440095 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2023.codfw.wmnet with OS book... [10:20:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440096 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS bookworm... [10:21:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440098 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS book... [10:34:20] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440137 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1267-1269].eqiad.wmnet} and (A:wiki... [10:36:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440145 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS bookworm... [10:37:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440147 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS book... [10:38:10] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440148 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from kubernetes1062 to wikikube-worker1087 completed: - ku... [10:42:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440166 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from kubernetes1061 to wikikube-worker1086 completed: - ku... [10:42:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440167 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from kubernetes1059 to wikikube-worker1084 completed: - ku... [10:47:05] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440173 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS bookworm... [10:47:35] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440174 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS book... [10:48:17] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440179 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jayme@cumin1002 from kubernetes1060 to wikikube-worker1085 completed: - ku... [10:51:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440182 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker1084.eqiad.wmnet with OS book... [10:51:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440200 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker1086.eqiad.wmnet with OS book... [10:52:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440203 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker1087.eqiad.wmnet with OS book... [10:52:34] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440204 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker1085.eqiad.wmnet with OS book... [11:06:23] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440232 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2023.codfw.wmnet with OS bookworm... [11:06:53] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440233 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS bookworm... [11:25:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440284 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS book... [11:31:48] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440298 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker1084.eqiad.wmnet with OS bookworm... [11:36:05] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440312 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker1085.eqiad.wmnet with OS bookworm... [11:40:35] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440318 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker1086.eqiad.wmnet with OS bookworm... [11:40:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440330 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker1087.eqiad.wmnet with OS bookworm... [11:41:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440331 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host wikikube-worker1086.eqiad.wmnet with OS book... [11:51:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440393 (10JMeybohm) [11:59:57] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 07Kubernetes: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T383213 (10JMeybohm) 03NEW [12:09:12] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440457 (10ops-monitoring-bot) pool host wikikube-worker[1084-1085,1087].eqiad.wmnet by jayme@cumin1002 with reason: None [12:09:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440458 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin1002 pool for host wikikube-worker[1084-1085,1087].eqia... [12:21:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440468 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host wikikube-worker1086.eqiad.wmnet with OS bookworm... [12:45:57] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440567 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS bookworm... [12:49:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440601 (10ops-monitoring-bot) pool host wikikube-worker2023.codfw.wmnet by jelto@cumin1002 with reason: None [12:49:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440602 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2023.codfw.wmnet comp... [12:50:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440621 (10ops-monitoring-bot) depool host wikikube-worker[2020-2021].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [12:51:46] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440632 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2020-2021].codfw.w... [12:52:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440634 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2021.codfw.wmnet with OS book... [12:52:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440635 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2020.codfw.wmnet with OS book... [12:53:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440638 (10ops-monitoring-bot) pool host wikikube-worker1086.eqiad.wmnet by jayme@cumin1002 with reason: None [12:53:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10440639 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin1002 pool for host wikikube-worker1086.eqiad.wmnet comp... [13:20:36] 06serviceops, 10Deployments, 06Release-Engineering-Team, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10440725 (10akosiaris) [Docs for our version](https://www.envoyproxy.io/docs/envoy/v1.23.12/configuration/http/http_filters/router_f... [13:39:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440827 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2020.codfw.wmnet with OS bookworm... [13:44:20] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440842 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2021.codfw.wmnet with OS bookworm... [13:50:21] 06serviceops, 10decommission-hardware: decommission mw13[49-51], mw1407 - https://phabricator.wikimedia.org/T383226 (10akosiaris) 03NEW [13:51:00] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10440903 (10akosiaris) [13:55:33] 06serviceops, 10decommission-hardware: decommission mw135[8-9], mw136[4-6], mw137[2-3], mw140[0-4], mw1406, mw14[11-13] - https://phabricator.wikimedia.org/T383227 (10akosiaris) 03NEW [13:56:42] 06serviceops, 10decommission-hardware: decommission mw13[49-51], mw1407 - https://phabricator.wikimedia.org/T383226#10440936 (10akosiaris) [13:57:53] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10440939 (10akosiaris) [14:06:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440964 (10Jelto) [14:07:57] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440965 (10ops-monitoring-bot) pool host wikikube-worker2021.codfw.wmnet by jelto@cumin1002 with reason: None [14:11:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440988 (10ops-monitoring-bot) pool host wikikube-worker2021.codfw.wmnet by jelto@cumin1002 with reason: None [14:11:20] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440989 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2021.codfw.wmnet comp... [14:11:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440991 (10ops-monitoring-bot) pool host wikikube-worker2020.codfw.wmnet by jelto@cumin1002 with reason: None [14:11:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440992 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2020.codfw.wmnet comp... [14:12:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10440996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS book... [14:35:52] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 13Patch-For-Review: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10441086 (10akosiaris) [14:48:48] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441170 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS bookworm... [15:06:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441257 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS book... [15:11:23] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441273 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS bookworm... [15:22:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441322 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS book... [15:40:43] 06serviceops, 06SRE: Framework for running experiments on a subset of the app server fleet - https://phabricator.wikimedia.org/T315403#10441452 (10lmata) Untagging sre-observability as we do some backlog housekeeping; please add us if we can assist. [15:51:26] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441504 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS bookworm... [15:51:52] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441507 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS book... [15:57:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441541 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2022.codfw.wmnet with OS bookworm... [16:01:01] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441554 (10ops-monitoring-bot) depool host wikikube-worker[2018-2019].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to bookworm + containerd [16:02:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441562 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2018-2019].codfw.w... [16:03:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441564 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2018.codfw.wmnet with OS book... [16:03:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441565 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2019.codfw.wmnet with OS book... [16:08:20] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T381878#10441587 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host w... [16:12:49] 06serviceops, 10Thumbor, 10Structured-Data-Backlog (Current Work): Reconsider use of `timeout` in Thumbor - https://phabricator.wikimedia.org/T379569#10441617 (10MarkTraceur) [16:25:14] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03), 13Patch-For-Review: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10441653 (10Scott_French) [16:25:23] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03), 13Patch-For-Review: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10441654 (10Scott_French) 05Open→03Resolved Alright, we now have logs in the... [16:43:54] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, and 2 others: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10441736 (10Jclark-ctr) @akosiaris i see you checked off all the boxes for the dcops team are these ready to be removed? [16:44:41] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441738 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2018.codfw.wmnet with OS bookworm... [16:50:51] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T381878#10441780 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikik... [16:52:07] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1081.eqiad.wmnet - https://phabricator.wikimedia.org/T381878#10441783 (10Jclark-ctr) @Jelto i performed flea power drain and looks to image properly the critical status h... [16:54:54] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Steady-state sizing of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T376519#10441793 (10Scott_French) Great, thank you both. I've updated the patch to reflect T376519#10439450. [16:56:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441794 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2019.codfw.wmnet with OS bookworm... [16:59:43] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441799 (10ops-monitoring-bot) pool host wikikube-worker2018.codfw.wmnet by jelto@cumin1002 with reason: None [16:59:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441800 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2018.codfw.wmnet comp... [16:59:52] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441801 (10ops-monitoring-bot) pool host wikikube-worker2019.codfw.wmnet by jelto@cumin1002 with reason: None [17:04:53] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441811 (10ops-monitoring-bot) pool host wikikube-worker2019.codfw.wmnet by jelto@cumin1002 with reason: None [17:04:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10441812 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker2019.codfw.wmnet comp... [17:31:12] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next - https://phabricator.wikimedia.org/T379986#10441918 (10Scott_French) @BPirkle - Many thanks for the testing effort and summary of your findi... [17:32:15] 06serviceops, 10Abstract Wikipedia team (25Q3 (Jan–Mar)): wikifunction httpbb tests fail because of title case issue - https://phabricator.wikimedia.org/T383032#10441920 (10Jdforrester-WMF) [17:36:27] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10441948 (10ops-monitoring-bot) depool host mw[1451-1455].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes [17:41:58] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10441992 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host mw[1451-1455].eqiad.wmnet com... [17:44:18] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10441999 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1453 to wikikube-worker1090 completed: - mw145... [17:44:21] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442001 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1453 to wikikube-worker1090 completed: - mw145... [17:49:39] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442031 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1451 to wikikube-worker1088 completed: - mw145... [17:53:37] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442044 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1453 to wikikube-worker1090 completed: - mw145... [17:57:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442063 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1452 to wikikube-worker1089 completed: - mw145... [18:04:50] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442097 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1454 to wikikube-worker1091 completed: - mw145... [18:09:16] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442141 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1455 to wikikube-worker1092 completed: - mw145... [18:11:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442147 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1088.eqiad.wmnet with OS... [18:12:30] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442149 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1089.eqiad.wmnet with OS... [18:13:00] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442162 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1090.eqiad.wmnet with OS... [18:13:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442174 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1091.eqiad.wmnet with OS... [18:13:43] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442175 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1092.eqiad.wmnet with OS... [18:52:14] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442350 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1088.eqiad.wmnet with OS bookworm completed: - wik... [18:55:23] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442382 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1092.eqiad.wmnet with OS bookworm completed: - wik... [19:01:17] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442418 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1090.eqiad.wmnet with OS bookworm completed: - wik... [19:03:09] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442424 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1091.eqiad.wmnet with OS bookworm completed: - wik... [19:08:32] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10442440 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1089.eqiad.wmnet with OS bookworm completed: - wik... [19:11:07] 06serviceops, 10Citoid, 10VisualEditor, 10VisualEditor-MediaWiki-References, and 2 others: Register Citoid as a "friendly bot" (or alternatively verified bot) with Cloudflare - https://phabricator.wikimedia.org/T370118#10442448 (10ppelberg) >>! In T370118#10437458, @akosiaris wrote: > Wooohooo. Thanks... [19:17:23] 06serviceops, 10decommission-hardware: decommission mw135[8-9], mw136[4-6], mw137[2-3], mw140[0-4], mw1406, mw14[11-13] - https://phabricator.wikimedia.org/T383227#10442476 (10ops-monitoring-bot) depool host wikikube-worker1001.eqiad.wmnet by kamila@cumin1002 with reason: decommission [19:17:56] 06serviceops, 10decommission-hardware: decommission mw135[8-9], mw136[4-6], mw137[2-3], mw140[0-4], mw1406, mw14[11-13] - https://phabricator.wikimedia.org/T383227#10442479 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host wikikube-worker1001.eqiad.w... [20:00:00] 06serviceops, 10decommission-hardware: decommission mw135[8-9], mw136[4-6], mw137[2-3], mw140[0-4], mw1406, mw14[11-13] - https://phabricator.wikimedia.org/T383227#10442617 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by kamila@cumin1002 for hosts: `wikikube-worker1001.eqiad.wmnet` - wikiku... [21:55:57] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10442873 (10akosiaris) [21:57:00] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10442874 (10akosiaris) >>! In T375842#10441736, @Jclark-ctr wrote: > @akosiaris i see you checked off all the boxes for the dcops team are these ready to be remove...