[08:04:41] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 4 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9807815 (10MoritzMuehlenhoff) [08:18:06] 06serviceops, 06Machine-Learning-Team, 13Patch-For-Review: Rename the envoy's uses_ingress option to sets_sni - https://phabricator.wikimedia.org/T346638#9807863 (10JMeybohm) [08:56:13] 06serviceops: ipoid charts app.job module has out of band changes - https://phabricator.wikimedia.org/T365224 (10JMeybohm) 03NEW [08:56:25] 06serviceops: ipoid charts app.job module has out of band changes - https://phabricator.wikimedia.org/T365224#9807912 (10JMeybohm) p:05Triage→03High [09:27:06] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9808056 (10MoritzMuehlenhoff) [09:33:37] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#9808067 (10BTullis) [10:36:40] jelto: Hej hej, if you have any thoughts to share regarding https://phabricator.wikimedia.org/T364839 (as I have no clue about requestctl) they'd be super welcome. And if not that's also cool, in that case I'll simply remove that downstream Phab code. Thanks in advance! [10:57:52] I'm out today, I'll check next week when I'm back [11:21:58] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9808280 (10MoritzMuehlenhoff) [11:51:44] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Co-locate kube-apiserver and etcd on new staging control plane nodes - https://phabricator.wikimedia.org/T363307#9808353 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d858a874-17ca-4ab5-8c9c-7fea35f1c823) set by jayme@... [12:12:45] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Co-locate kube-apiserver and etcd on new staging control plane nodes - https://phabricator.wikimedia.org/T363307#9808366 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jayme@cumin1002 for hosts: `kubestagemaster[1001-10... [12:24:58] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Co-locate kube-apiserver and etcd on new staging control plane nodes - https://phabricator.wikimedia.org/T363307#9808396 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=dd087345-70da-428c-8704-76433fe47872) set by jayme@... [12:57:01] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Co-locate kube-apiserver and etcd on new staging control plane nodes - https://phabricator.wikimedia.org/T363307#9808481 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jayme@cumin1002 for hosts: `kubestagetcd[1004-1006]... [12:58:43] akosiaris: 👋 We are planning to upgrade mobileapps to node18. Do we need a patch similar to this one in order for ipv6 priority to work or mobileapps is ready? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1023824 [13:09:21] nemo-yiannis: yes we do [13:09:28] want me to handle it? [13:09:38] if you can? i am not sure what needs to happen [13:09:48] Let me file a ticket under the nodejs upgrade [13:12:07] akosiaris: care to piggyback https://phabricator.wikimedia.org/T362978 ? [13:12:43] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 9 others: Enable ipv6 in mesh used in PCS k8s deployment - https://phabricator.wikimedia.org/T365250 (10Jgiannelos) 03NEW [13:18:37] jayme: done in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1032779 [13:18:55] <3 - will check in 5' [13:19:16] there's so many Bug lines in that patch that jenkins is going to yell some profanity my way I guess [13:19:31] eheh [13:21:18] did not happen apparently [13:22:48] btw. akosiaris I tried (not very hard) to understand why we duplicate listeners for v4 and v6 instead of enabling compat. Do you have a tl;dr by chance? [13:23:09] the TL;DR is that compact sucks [13:23:15] compat* [13:23:21] ahah [13:23:35] it was conceived as an idea to save on resources, where resources were sockets [13:24:03] BSDs never defaulted to it (although they have implemented it) [13:24:08] linux did default to it [13:24:38] but the rough thing is that embedded IPv4 in IPv6 address are confusing, not just to humans but to programs too [13:27:43] ok [13:28:10] a larger version is in https://phabricator.wikimedia.org/T255568#6779477 [13:28:34] and the en-following discussion between me and jbon.d [13:31:41] I will (re-)read. Thanks [13:37:40] 06serviceops, 10Citoid, 06Content-Transform-Team-WIP, 10CX-cxserver, and 7 others: Upgrade mobileapps to node 18 - https://phabricator.wikimedia.org/T363168#9808721 (10Lucas_Werkmeister_WMDE) [13:37:45] 06serviceops, 10Citoid, 06Content-Transform-Team-WIP, 10CX-cxserver, and 7 others: Enable ipv6 in mesh used in PCS k8s deployment - https://phabricator.wikimedia.org/T365250#9808722 (10Lucas_Werkmeister_WMDE) [13:49:42] 06serviceops, 06Machine-Learning-Team, 07Kubernetes: Allow Kubernetes workers to be deployed on Bookworm - https://phabricator.wikimedia.org/T365253 (10elukey) 03NEW [13:50:37] 06serviceops, 06Machine-Learning-Team, 07Kubernetes: Allow Kubernetes workers to be deployed on Bookworm - https://phabricator.wikimedia.org/T365253#9808792 (10elukey) [13:53:40] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Co-locate kube-apiserver and etcd on new staging control plane nodes - https://phabricator.wikimedia.org/T363307#9808799 (10JMeybohm) [13:58:46] 06serviceops, 06Machine-Learning-Team, 07Kubernetes: Allow Kubernetes workers to be deployed on Bookworm - https://phabricator.wikimedia.org/T365253#9808809 (10JMeybohm) For {T362408} we're planning to backport containerd from bookworm to bullseye. Maybe it would be feasible to backport runc as well (althoug... [14:03:22] 06serviceops, 06Machine-Learning-Team, 07Kubernetes: Allow Kubernetes workers to be deployed on Bookworm - https://phabricator.wikimedia.org/T365253#9808825 (10elukey) ML would be very happy to test the 6.x kernel since the GPU drivers are shipped directly with it, so we'd get a nice bump to those as well. I... [14:23:32] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Co-locate kube-apiserver and etcd on new staging control plane nodes - https://phabricator.wikimedia.org/T363307#9808855 (10JMeybohm) 05Open→03Resolved Both staging clusters have been migrated to stacked control-planes [15:10:30] 06serviceops, 10Cassandra, 06SRE, 10Data Products (Data Products Sprint 13), and 2 others: Commons Impact Metrics: Data Gateway endpoints - https://phabricator.wikimedia.org/T364921#9809043 (10Eevans) >>! In T364921#9807379, @Scott_French wrote: > Many thanks for getting the image builds running and settin... [15:11:08] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube control planes to hardware nodes - https://phabricator.wikimedia.org/T353464#9809039 (10JMeybohm) [15:19:58] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 10SRE Observability (FY2023/2024-Q4): Create a per-release deployment of statsd-exporter for mw-on-k8s - https://phabricator.wikimedia.org/T365265 (10Clement_Goubert) 03NEW [15:20:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube control planes to hardware nodes - https://phabricator.wikimedia.org/T353464#9809116 (10hnowlan) [15:20:56] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2023/2024-Q4): Create a per-release deployment of statsd-exporter for mw-on-k8s - https://phabricator.wikimedia.org/T365265#9809110 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [15:21:08] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9809121 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye [15:21:31] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2023/2024-Q4): Create a per-release deployment of statsd-exporter for mw-on-k8s - https://phabricator.wikimedia.org/T365265#9809123 (10Clement_Goubert) [16:17:44] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9809368 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed... [18:40:32] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9809808 (10Jhancock.wm) a:05Jhancock.wm→03Papaul @Papaul I'm still having trouble with the same spot as noted before. Can you take a look at it? [19:10:28] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9809946 (10Papaul) Thank you will do [19:22:00] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9810014 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye [19:42:42] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9810061 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed wi... [19:43:21] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9810065 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye [21:10:51] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9810270 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed wi... [21:47:25] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9810341 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main1006.eqiad.wmn... [21:57:32] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9810363 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main1006.eqiad.wmnet w... [21:58:12] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9810364 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main1006.eqiad.wmn... [22:20:24] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9810418 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet wi... [22:24:01] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9810421 (10akosiaris) [22:26:27] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9810438 (10akosiaris) For some reason on kafka1006 software RAID re-syncing is taking forever (moving at 19K/s, which is VERY slow) and... [22:44:02] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9810468 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main1006.eqiad.wmnet with OS bullseye execut... [23:08:16] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9810496 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed wi... [23:41:40] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9810548 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye