[01:50:43] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye [01:50:47] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye [01:50:49] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2037.codfw.wmnet with OS bullseye [02:37:15] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Jhancock.wm) [08:13:24] hello folks [08:13:33] proceeding with kafka main eqiad reboots [08:47:52] <3 [08:54:43] folks I am thinking to postpone any kafka main rebalance to after the switchover, to avoid any issues [08:55:15] one thing to keep an eye on, when we switch, is how the new traffic pattern in kafka-main affects its metrics [08:57:49] 10serviceops, 10Prod-Kubernetes: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507 (10JMeybohm) p:05Low→03High [11:32:12] 10serviceops, 10Similarusers: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274 (10hnowlan) [11:51:20] 10serviceops, 10Observability-Tracing: Add ingress to aux-k8s - https://phabricator.wikimedia.org/T325178 (10JMeybohm) a:05Clement_Goubert→03JMeybohm [11:55:26] kafka reboots done [11:56:25] gg elukey, thanks so much [12:41:54] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Datacenter-Switchover: CommRel support for September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345265 (10kamila) [12:41:58] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Datacenter-Switchover: CommRel support for September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345265 (10kamila) [12:45:21] 10serviceops, 10Observability-Tracing, 10Patch-For-Review: Add ingress to aux-k8s - https://phabricator.wikimedia.org/T325178 (10JMeybohm) [12:49:07] 10serviceops, 10Observability-Tracing, 10Patch-For-Review: Add ingress to aux-k8s - https://phabricator.wikimedia.org/T325178 (10JMeybohm) 05In progress→03Resolved Did the LVS dance, `curl -4 --resolve jaeger-query.discovery.wmnet:30443:$(dig +short k8s-ingress-aux.svc.eqiad.wmnet) https://jaeger-query.d... [12:49:10] 10serviceops, 10Observability-Tracing, 10Patch-For-Review, 10User-fgiunchedi: jaeger is configured to receive traces from production - https://phabricator.wikimedia.org/T344253 (10JMeybohm) [12:51:09] 10serviceops, 10Observability-Tracing, 10Patch-For-Review, 10User-fgiunchedi: jaeger is configured to receive traces from production - https://phabricator.wikimedia.org/T344253 (10JMeybohm) [14:10:38] 10serviceops, 10Parsoid (Tracking): Upgrade nodejs on testreduce1001 - https://phabricator.wikimedia.org/T345220 (10MSantos) [14:42:32] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10VRiley-WMF) kubernetes1031 - A 6. U 37. port 30 Cableid 4017 kubernetes1030 - A 6. U 36. port 31 Cableid 1917 kubernetes1029 - A 6. U 35. port24 Cableid: 1947 [15:02:21] 10serviceops, 10Wikimedia-Site-requests, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint): Configure the aggregation job to run periodically on Wikimedia wikis - https://phabricator.wikimedia.org/T339984 (10Daimona) Hi #serviceops! I've added this tag based on T320403. Please let u... [15:37:23] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Datacenter-Switchover: CommRel support for September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345265 (10Trizek-WMF) a:03Trizek-WMF @kamila, thank you for asking for our support. We have a message ready for commu... [15:37:40] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Datacenter-Switchover: CommRel support for September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345265 (10Trizek-WMF) p:05Triage→03High [16:31:02] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Datacenter-Switchover: CommRel support for September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345265 (10RZamora-WMF) Thanks for claiming this Phab task 👍 [16:43:39] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Datacenter-Switchover: CommRel support for September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345265 (10kamila) Thank you @Trizek-WMF ! The message looks good. Maybe I'd suggest replacing the word "first" with "prim... [20:00:04] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye [20:00:08] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye [20:00:14] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye executed with errors: - kubernetes... [20:00:20] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye executed with errors: - kubernetes... [20:00:26] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2037.codfw.wmnet with OS bullseye [20:01:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye [20:01:52] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye executed with errors: - kubernetes... [20:11:13] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye [20:11:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye executed with errors: - kubernetes... [20:13:37] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye [20:13:46] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye executed with errors: - kubernetes... [20:20:48] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye [20:20:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye executed with errors: - kubernetes... [21:13:36] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubernetes2037.codfw.wmnet with OS bullseye executed with errors: - kubernetes...