[10:13:04] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10496145 (10jijiki) [10:17:09] 06serviceops, 10MW-on-K8s: Migrate mwmaint server functionality to mw-on-k8s - https://phabricator.wikimedia.org/T341560#10496170 (10Lucas_Werkmeister_WMDE) I’m not sure if it should be a subtask of this or not, but if we’re planning to sunset the mwmaint servers, see also: {T384764} [10:24:24] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10496206 (10jijiki) [10:46:33] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 10Maps (Kartotherian): Staging error: SVG attribute is not supported - https://phabricator.wikimedia.org/T384823 (10Jgiannelos) 03NEW [10:55:23] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10496373 (10jijiki) [11:04:47] 06serviceops, 10Observability-Logging, 07Kubernetes: Move rsyslog-generated logs within k8s to their own kafka topics - https://phabricator.wikimedia.org/T384335#10496412 (10fgiunchedi) + serviceops / k8s for visibility. e.g. @JMeybohm please let me know what you think [11:10:49] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10496454 (10Lucas_Werkmeister_WMDE) [11:13:59] 06serviceops, 10MW-on-K8s, 10Observability-Logging, 07Kubernetes: Move rsyslog-generated logs within k8s to their own kafka topics - https://phabricator.wikimedia.org/T384335#10496467 (10JMeybohm) I'd assume these are mediawiki generated logs, but I don't think there is an rsyslog daemonset. Do you have so... [11:14:20] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10496470 (10jijiki) [11:23:02] 06serviceops, 10MW-on-K8s, 10Observability-Logging, 07Kubernetes: Move rsyslog-generated logs within k8s to their own kafka topics - https://phabricator.wikimedia.org/T384335#10496505 (10fgiunchedi) [11:34:33] 06serviceops, 10MW-on-K8s, 10Observability-Logging, 07Kubernetes: Move rsyslog-generated logs within k8s to their own kafka topics - https://phabricator.wikimedia.org/T384335#10496560 (10fgiunchedi) You are correct these are mw logs only, and no daemonset but a sidecar AFAICS. Example log: `lines=4,lang=j... [11:37:47] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10496573 (10MoritzMuehlenhoff) [11:41:03] 06serviceops, 10MW-on-K8s, 10Observability-Logging, 07Kubernetes: Move rsyslog-generated mediawiki logs within k8s to their own kafka topics - https://phabricator.wikimedia.org/T384335#10496607 (10fgiunchedi) [11:52:22] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10496659 (10Papaul) [11:53:22] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10496664 (10Papaul) @Jhancock.wm you can move ganeti2020 anytime today. Once done just ping @MoritzMuehlenho... [12:24:44] effie: o/ something that it is not clear to me from your PHP8.1 email - is the base OS image being bumped too? To say bookworm [12:35:29] ah that is a fair point! PHP8.1 images are based on bullseye https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1064814 [12:46:01] 06serviceops, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07Wikimedia-Performance-recommendation: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10496752 (10larissagaulia) [12:46:50] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 10Maps (Kartotherian): Staging error: SVG attribute is not supported - https://phabricator.wikimedia.org/T384823#10496753 (10elukey) I found this issue that may be interesting, it highlights the same errors. Somebody points to: [13:06:26] 06serviceops: create deployment.discovery.wmnet or figure out why we don't have it - https://phabricator.wikimedia.org/T384736#10496800 (10jijiki) p:05Triage→03Low [13:10:34] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 10Maps (Kartotherian): Staging error: SVG attribute is not supported - https://phabricator.wikimedia.org/T384823#10496808 (10Jgiannelos) Unfortunately even clearing the errors the rendered labels are still broken. [13:11:11] effie: okok nice! So we'll also clean up old buster images, really good :) [13:11:28] at some point we'll also have to migrate to bookworm :) [13:24:09] needs another ICU migration before, though [13:33:08] one step at a time folks:) [13:44:22] 06serviceops, 10observability, 10Observability-Metrics: Repeated library panels showing only after refresh, not on first load - https://phabricator.wikimedia.org/T384831 (10jijiki) 03NEW p:05Triage→03High [13:48:19] 06serviceops, 10observability, 10Observability-Metrics: Repeated library panels showing only after refresh, not on first load - https://phabricator.wikimedia.org/T384831#10496932 (10jijiki) [13:48:56] 06serviceops, 10observability, 10Observability-Metrics: Repeated library panels in Grafana showing only after refresh, not on first load - https://phabricator.wikimedia.org/T384831#10496939 (10jijiki) [13:55:33] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 10Maps (Kartotherian): Staging error: SVG attribute is not supported - https://phabricator.wikimedia.org/T384823#10496954 (10Jgiannelos) Update: If I force a specific sans font things are rendered better. I think that the errors are just... [14:28:23] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497104 (10ops-monitoring-bot) depool host parse[1018-1024].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes [14:32:29] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497128 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host parse[1018-1024].eqiad.wmnet... [14:39:40] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497161 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1018 to wikikube-worker1159 completed: - pa... [14:46:01] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497203 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1019 to wikikube-worker1160 completed: - pa... [14:49:34] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497208 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1020 to wikikube-worker1161 completed: - pa... [14:53:35] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497246 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1021 to wikikube-worker1162 completed: - pa... [14:53:42] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 06SRE: wikikube-ctrl1002 and wikikube-ctrl1003: Switch network cable from port 2 to port 1 on the 10G NIC - https://phabricator.wikimedia.org/T379717#10497248 (10Papaul) @JMeybohm can we do this today? if not please let me know when will be a good d... [14:56:27] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T383620#10497264 (10kamila) a:03VRiley-WMF [14:58:16] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497272 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1022 to wikikube-worker1163 completed: - pa... [15:02:05] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497289 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1023 to wikikube-worker1164 completed: - pa... [15:07:23] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497317 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1024 to wikikube-worker1165 completed: - pa... [15:09:02] 06serviceops, 10observability, 10Observability-Metrics: Repeated library panels in Grafana showing only after refresh, not on first load - https://phabricator.wikimedia.org/T384831#10497341 (10fgiunchedi) Thank you for the bug report and for finding the upstream issue; I've read through it and a repeat/dupli... [15:09:48] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 06SRE: wikikube-ctrl1002 and wikikube-ctrl1003: Switch network cable from port 2 to port 1 on the 10G NIC - https://phabricator.wikimedia.org/T379717#10497343 (10ops-monitoring-bot) depool host wikikube-ctrl[1002-1003].eqiad.wmnet by jayme@cumin1002... [15:10:01] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 06SRE: wikikube-ctrl1002 and wikikube-ctrl1003: Switch network cable from port 2 to port 1 on the 10G NIC - https://phabricator.wikimedia.org/T379717#10497344 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin1... [15:10:43] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497347 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1160.eqiad.wmnet with OS... [15:10:44] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497348 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1161.eqiad.wmnet with OS... [15:10:55] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497349 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1162.eqiad.wmnet with OS... [15:11:09] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497353 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1163.eqiad.wmnet with OS... [15:11:12] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497354 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1164.eqiad.wmnet with OS... [15:11:16] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497355 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1165.eqiad.wmnet with OS... [15:11:45] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497357 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1159.eqiad.wmnet with OS... [15:43:14] 06serviceops, 10decommission-hardware: decommission mw135[8-9], mw136[4-6], mw137[2-3], mw140[0-4], mw1406, mw14[11-13] - https://phabricator.wikimedia.org/T383227#10497445 (10akosiaris) [15:53:19] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497490 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1160.eqiad.wmnet with OS boo... [15:54:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497499 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1161.eqiad.wmnet with OS boo... [15:58:22] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497514 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1162.eqiad.wmnet with OS boo... [16:02:06] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497540 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1159.eqiad.wmnet with OS boo... [16:05:27] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497557 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1165.eqiad.wmnet with OS boo... [16:08:43] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497581 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1164.eqiad.wmnet with OS boo... [16:09:02] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497582 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1163.eqiad.wmnet with OS boo... [16:09:56] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10497584 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1164.eqiad.wmnet with OS... [20:36:37] 06serviceops, 10MW-on-K8s, 06SRE: mwgrep cannot be used from a deployment host - https://phabricator.wikimedia.org/T384764#10498755 (10RLazarus) This isn't working because it was never upgraded to Python 3. (`reload` was a built-in function in Python 2, moved to `importlib` in 3.) The mwmaint hosts are still... [22:31:38] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 06SRE: wikikube-ctrl1002 and wikikube-ctrl1003: Switch network cable from port 2 to port 1 on the 10G NIC - https://phabricator.wikimedia.org/T379717#10499096 (10VRiley-WMF) Is this okay to be closed?