[06:50:52] 06serviceops, 06MediaWiki-Platform-Team, 13Patch-For-Review: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10482745 (10Joe) >>! In T384294#10481694, @Scott_French wrote: > Ah, interesting! I was unaware of the changes in 8.3 mentioned in T384294#10480405. If that's the case... [08:47:01] 06serviceops, 10MW-on-K8s, 10Observability-Logging: Unexpected utilization increase in udp_localhost-info kafka-logging topic - https://phabricator.wikimedia.org/T384233#10482872 (10fgiunchedi) 05Stalled→03Resolved a:03fgiunchedi Topic retention now works properly for udp_localhost-info, resolving [09:21:05] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10482944 (10JMeybohm) [09:39:36] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10483046 (10JMeybohm) [09:43:03] 06serviceops, 06Abstract Wikipedia team, 10Prod-Kubernetes, 10Wikifunctions, 07Kubernetes: Migrate deprecated apparmor.security.beta.kubernetes.io annotations to SecurityContext - https://phabricator.wikimedia.org/T384429 (10JMeybohm) 03NEW [09:44:30] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10483058 (10JMeybohm) [09:55:19] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 07Epic, 10Maps (Kartotherian): Staging error: Empty feature collection on geojson raises error - https://phabricator.wikimedia.org/T384435 (10Jgiannelos) 03NEW [09:57:46] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Staging error: Empty feature collection on geojson raises error - https://phabricator.wikimedia.org/T384435#10483155 (10Jgiannelos) [11:12:44] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update wikikube-staging-codfw to kubernetes 1.31 - https://phabricator.wikimedia.org/T384450 (10JMeybohm) 03NEW [11:13:54] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 3 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10483483 (10Marostegui) @Papaul db2166 can be moved anytime. The host has been powered off. [11:43:39] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10483614 (10Jelto) >>! In T383709#10480819, @Papaul wrote: > @Jelto thanks please let us know when best work... [12:11:28] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian), 13Patch-For-Review: Staging error: Empty feature collection on geojson raises error - https://phabricator.wikimedia.org/T384435#10483692 (10Jgiannelos) a:03Jgiannelos [12:22:53] 06serviceops, 06MediaWiki-Platform-Team, 13Patch-For-Review: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10483732 (10jijiki) >>! In T384294#10482745, @Joe wrote: >>>! In T384294#10481694, @Scott_French wrote: >> Ah, interesting! I was unaware of the changes in 8.3 mention... [12:44:41] 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 2 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10483801 (10hnowlan) A test rollout of the routing for testwiki was successful when testing internally from a routing perspective,... [13:40:14] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10483932 (10JMeybohm) [13:46:27] 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 2 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10483956 (10Mvolz) >>! In T361576#10483801, @hnowlan wrote: > A test rollout of the routing for testwiki was successful when test... [13:53:30] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10483996 (10Jhancock.wm) @Marostegui i'll move it this morning. @Jelto That time works for us. thank you... [14:00:04] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10484020 (10Jhancock.wm) [15:29:14] 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 2 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10484518 (10hnowlan) >>! In T361576#10483956, @Mvolz wrote: > >>>! In T361576#10483801, @hnowlan wrote: >> A test rollout of the... [15:57:03] 06serviceops, 10MediaWiki-extensions-OAuth, 10MediaWiki-Platform-Team (Roadmap), 07Technical-Debt, 07Upstream: Migrate OAuth extension back from wikimedia/oauth2-server fork to upstream - https://phabricator.wikimedia.org/T261462#10484808 (10JTweed-WMF) [15:59:29] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10484874 (10ops-monitoring-bot) depool host mw[1476-1481].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes [16:04:16] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10484942 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host mw[1476-1481].eqiad.wmnet com... [16:11:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10484965 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1476 to wikikube-worker1129 completed: - mw147... [16:15:35] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10484982 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1477 to wikikube-worker1130 completed: - mw147... [16:19:56] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485023 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1478 to wikikube-worker1131 completed: - mw147... [16:20:00] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10485024 (10Jhancock.wm) @Marostegui db2166 is moved, updated, and pinging. [16:20:25] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10485035 (10Jhancock.wm) [16:24:30] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485051 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1479 to wikikube-worker1132 completed: - mw147... [16:28:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485085 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1480 to wikikube-worker1133 completed: - mw148... [16:33:09] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485123 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1481 to wikikube-worker1134 completed: - mw148... [16:36:06] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485140 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1129.eqiad.wmnet with OS... [16:36:13] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485141 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1130.eqiad.wmnet with OS... [16:36:21] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1131.eqiad.wmnet with OS... [16:36:26] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485143 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1132.eqiad.wmnet with OS... [16:36:36] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485144 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1133.eqiad.wmnet with OS... [16:36:39] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485145 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1134.eqiad.wmnet with OS... [16:52:54] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485279 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1134.eqiad.wmnet with OS boo... [16:53:14] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485285 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1134.eqiad.wmnet with OS... [17:14:29] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485392 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1129.eqiad.wmnet with OS boo... [17:14:57] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485395 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1130.eqiad.wmnet with OS boo... [17:16:30] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485407 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1130.eqiad.wmnet with OS... [17:18:02] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485410 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1131.eqiad.wmnet with OS boo... [17:21:49] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485426 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1133.eqiad.wmnet with OS boo... [17:26:24] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485466 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1132.eqiad.wmnet with OS boo... [17:31:53] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10485522 (10jijiki) 05Open→03In progress [17:32:25] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485532 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1134.eqiad.wmnet with OS boo... [17:36:20] 06serviceops, 06MediaWiki-Platform-Team, 13Patch-For-Review: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10485566 (10jijiki) p:05Triage→03Low [17:41:35] 06serviceops, 10Prod-Kubernetes, 10Wikifunctions, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 07Kubernetes: Migrate deprecated apparmor.security.beta.kubernetes.io annotations to SecurityContext - https://phabricator.wikimedia.org/T384429#10485644 (10Jdforrester-WMF) [17:44:31] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10485666 (10Jhancock.wm) [17:56:48] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485724 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1130.eqiad.wmnet with OS boo... [17:59:02] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T383620#10485738 (10kamila) [18:12:46] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485822 (10ops-monitoring-bot) pool host wikikube-worker[1129-1134].eqiad.wmnet by kamila@cumin1002 with reason: None [18:12:48] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10485823 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1129-1134].eqia... [18:41:29] 06serviceops: Support cookie-driven fractional migration to PHP 8.1 deployments of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T377042#10485935 (10Scott_French) 05Open→03Resolved In mwdebug and then subsequently after wider deployment, I was able to verify that: 1. In the absence of `PHP_EN... [19:37:15] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Difftesting between staging and production - https://phabricator.wikimedia.org/T384530 (10Jgiannelos) 03NEW [19:44:22] 06serviceops: Align mw-on-k8s alerts with PHP 8.1 migration - https://phabricator.wikimedia.org/T384532 (10Scott_French) 03NEW [19:54:22] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Difftesting between staging and production - https://phabricator.wikimedia.org/T384530#10486238 (10Jgiannelos) # I created a dataset with Kartotherian URLs after parsing wikipedia articles that have kartotherian re... [20:17:40] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486339 (10ops-monitoring-bot) depool host mw[1482-1488].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes [20:21:37] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486346 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host mw[1482-1488].eqiad.wmnet com... [20:36:36] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486367 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1482 to wikikube-worker1135 completed: - mw148... [20:40:58] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486379 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1483 to wikikube-worker1136 completed: - mw148... [20:45:15] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486407 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1484 to wikikube-worker1137 completed: - mw148... [20:49:28] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486411 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1485 to wikikube-worker1138 completed: - mw148... [20:54:02] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486431 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1486 to wikikube-worker1139 completed: - mw148... [20:58:59] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486449 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1487 to wikikube-worker1140 completed: - mw148... [21:02:45] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486470 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1488 to wikikube-worker1141 completed: - mw148... [21:03:36] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486475 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1136.eqiad.wmnet with OS... [21:03:42] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486477 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1137.eqiad.wmnet with OS... [21:03:50] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486479 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1138.eqiad.wmnet with OS... [21:04:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486480 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1140.eqiad.wmnet with OS... [21:04:25] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486481 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1139.eqiad.wmnet with OS... [21:04:29] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486483 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1141.eqiad.wmnet with OS... [21:04:51] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486485 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1135.eqiad.wmnet with OS... [21:42:22] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486622 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1136.eqiad.wmnet with OS boo... [21:45:21] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486636 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1141.eqiad.wmnet with OS boo... [21:49:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486659 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1135.eqiad.wmnet with OS boo... [21:52:37] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486667 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1140.eqiad.wmnet with OS boo... [21:56:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486694 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1137.eqiad.wmnet with OS boo... [21:59:52] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486706 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1138.eqiad.wmnet with OS boo... [22:03:41] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486709 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1139.eqiad.wmnet with OS boo... [22:20:15] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486784 (10ops-monitoring-bot) pool host wikikube-worker[1135-1141].eqiad.wmnet by kamila@cumin1002 with reason: None [22:20:18] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10486785 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1135-1141].eqia... [22:27:22] 06serviceops, 10MW-on-K8s, 10Observability-Logging: Unexpected utilization increase in udp_localhost-info kafka-logging topic - https://phabricator.wikimedia.org/T384233#10486798 (10colewhite) I'm curious, is it a known issue with our version of kafka that the retention settings need wiggled occasionally... [23:01:46] 06serviceops, 10Cassandra, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10486942 (10Eevans) >>! In T368096#10377649, @Eevans wrote: >>>! In T368096#10046564, @Eevans wrote: >> @Cparle how hard would it be to create some functional te...