[00:30:22] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10460814 (10Scott_French) [01:04:02] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next - https://phabricator.wikimedia.org/T379986#10460856 (10Scott_French) @MSantos - Thanks for the heads-up! Just to make sure I understand, it... [03:03:54] 06serviceops, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), 10MW-Interfaces-Team (MWI-Sprint-1 (2025-01-14 to 2025-01-28)), 07User-notice: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title end... - https://phabricator.wikimedia.org/T374683#10460920 [05:07:15] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10460982 (10Aklapper) [Please review subscribers and project tags when creating a subtask - thanks] [06:50:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461039 (10ops-monitoring-bot) depool host mw[2369-2372].codfw.wmnet by jelto@cumin1002 with reason: Renaming nodes [06:52:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461041 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host mw[2369-2372... [07:06:23] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461046 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2369 to wikikube-worker2220 c... [07:38:38] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461064 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2370 to wikikube-worker2221 completed: - mw2370 (**... [07:51:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461078 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2371 to wikikube-worker2222 completed: - mw2371 (**... [08:00:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461081 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2372 to wikikube-worker2223 completed: - mw2372 (**... [08:10:12] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461088 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jelto@cumin1002 Renumbering for host wikikube-worker2220.codfw.wm... [08:10:24] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461089 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2220.codfw.wmnet with OS bull... [08:18:34] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461098 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jelto@cumin1002 Renumbering for host wikikube-worker2221.codfw.wm... [08:18:53] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461113 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2221.codfw.wmnet with OS bull... [08:25:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461129 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jelto@cumin1002 Renumbering for host wikikube-worker2222.codfw.wm... [08:25:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461130 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2222.codfw.wmnet with OS bull... [08:31:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461155 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node was started by jelto@cumin1002 Renumbering for host wikikube-worker2223.codfw.wm... [08:31:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461156 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2223.codfw.wmnet with OS bull... [08:31:58] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Move kafka-main2010 within the same rack - https://phabricator.wikimedia.org/T381788#10461157 (10JMeybohm) >>! In T381788#10458970, @Jhancock.wm wrote: > We are off on the 20th in the US. but the rest of the week is good for me. Sorry, I wasn't aware. What abou... [08:53:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461200 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2220.codfw.wmnet with OS bullseye... [09:03:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461229 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2221.codfw.wmnet with OS bullseye... [09:10:38] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461250 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2222.codfw.wmnet with OS bullseye... [09:11:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461254 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jelto@cumin1002 Renumbering for host wikikube-worker2220.codfw.wmnet... [09:16:02] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461264 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2223.codfw.wmnet with OS bullseye... [09:30:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461289 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jelto@cumin1002 Renumbering for host wikikube-worker2221.codfw.wmnet... [09:33:05] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10461298 (10JMeybohm) I thought it would help to add a bit more details on how to reproduce (like a complete curl, so others don't have... [09:38:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461322 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jelto@cumin1002 Renumbering for host wikikube-worker2222.codfw.wmnet... [09:45:01] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461349 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.renumber-node started by jelto@cumin1002 Renumbering for host wikikube-worker2223.codfw.wmnet... [09:51:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461360 (10ops-monitoring-bot) depool host wikikube-worker[2220-2223].codfw.wmnet by jelto@cumin1002 with reason: Renaming nodes [09:51:49] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461361 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-worker[2220-2223].codfw.w... [09:55:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461384 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2220.codfw.wmnet with OS book... [09:59:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461418 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2221.codfw.wmnet with OS book... [10:00:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461420 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2222.codfw.wmnet with OS book... [10:01:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461428 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2223.codfw.wmnet with OS book... [10:17:26] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10461487 (10Jgiannelos) So i did some debugging and i found out that here is the codebase that kartotherian calls the external endpoint... [10:36:30] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461531 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2220.codfw.wmnet with OS bookworm... [10:42:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461555 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2221.codfw.wmnet with OS bookworm... [10:44:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461564 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2223.codfw.wmnet with OS bookworm... [10:48:03] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10461597 (10Jgiannelos) Hm, it doesn't use the regular pattern with the request template configurable that we have in the rest of the se... [10:53:00] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461615 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2222.codfw.wmnet with OS bookworm... [11:15:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461685 (10ops-monitoring-bot) pool host wikikube-worker[2220-2223].codfw.wmnet by jelto@cumin1002 with reason: None [11:15:24] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461686 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[2220-2223].codfw.wmn... [11:16:26] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T383764 (10Jelto) 03NEW [12:30:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461833 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host mw[2359,2366... [12:44:52] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10461872 (10LucasWerkmeister) >>! In T380544#10346771, @LucasWerkmeister wrote: > To put a concrete number out: at the moment, Quarry reports [79158475 links](https://quarry.... [12:45:17] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2359 to wikikube-worker2224 c... [12:52:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461893 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2366 to wikikube-worker2225 c... [12:56:48] Hi, Does this change look good to serviceops? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1109526 [12:58:26] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10461903 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2367 to wikikube-worker2226 c... [13:16:42] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10461961 (10JMeybohm) >>! In T383710#10461487, @Jgiannelos wrote: > From what I understand we need to change kartotherian so that it can... [13:26:44] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10461996 (10elukey) Reporting some slack conversations (maps channel): I can confirm it is a call to the MWAPI the issue: ` tcp 0... [13:31:00] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462016 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2368 to wikikube-worker2227 c... [13:43:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462100 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2224.codfw.wmnet with OS book... [13:47:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462124 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2225.codfw.wmnet with OS book... [13:48:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462125 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2226.codfw.wmnet with OS book... [13:52:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462132 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2227.codfw.wmnet with OS book... [14:25:13] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10462224 (10jijiki) [14:29:12] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10462250 (10jijiki) shellbox-syntaxhighlight PHP 8.1 has been rolled out to codfw as well, so far things look alright :) * https://grafana.wikimedia.org/goto/yptnXsvHg?orgId=1 * https:... [14:30:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462257 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2224.codfw.wmnet with OS bookworm... [14:30:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462258 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2225.codfw.wmnet with OS bookworm... [14:30:23] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462259 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2226.codfw.wmnet with OS bookworm... [14:30:44] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462261 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2224.codfw.wmnet with OS book... [14:30:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462263 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2225.codfw.wmnet with OS book... [14:31:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462265 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2226.codfw.wmnet with OS book... [14:42:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462387 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2227.codfw.wmnet with OS bookworm... [14:47:44] 06serviceops, 10MW-on-K8s, 06Traffic, 13Patch-For-Review: Some sites try and fail to serve favicon.ico - https://phabricator.wikimedia.org/T374997#10462412 (10matmarex) While testing the changes, we realized that the icons were previously also accessible under URLs like https://meta.wikimedia.org/w/favicon... [14:49:31] 06serviceops, 10MW-on-K8s, 06Traffic: Some sites try and fail to serve favicon.ico - https://phabricator.wikimedia.org/T374997#10462447 (10matmarex) 05Open→03Resolved a:05akosiaris→03matmarex URLs in the task description now work correctly. [14:55:30] 06serviceops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE-tools, 10Sustainability (Incident Followup): Write a cookbook to set a k8s cluster in maintenance mode - https://phabricator.wikimedia.org/T277677#10462482 (10elukey) a:05elukey→03None [14:58:08] 06serviceops: operations/docker-images/production-images contains references to non-existent image python3 - https://phabricator.wikimedia.org/T336682#10462498 (10elukey) 05Open→03Resolved @hashar sorry for the gigantic delay but I missed the update. IIRC I went for https://wikitech.wikimedia.org/wiki/Do... [14:58:31] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Move kafka-main2010 within the same rack - https://phabricator.wikimedia.org/T381788#10462502 (10Jhancock.wm) that will work for us. [15:05:41] 06serviceops, 06Infrastructure-Foundations, 06Machine-Learning-Team: Migrate the ownership of Docker images in production-images repo to mailing lists - https://phabricator.wikimedia.org/T373526#10462535 (10elukey) To summarize the current state: Single owner: ` cert-manager/build/control: Janis Meybohm 06serviceops, 06Infrastructure-Foundations, 07Security: Upgrade K8s docker images running in Wikimedia production on Buster to either Bullseye or Bookworm - https://phabricator.wikimedia.org/T368366#10462548 (10elukey) 05Open→03Resolved Most of the images have been migrated, we have some subtasks sti... [15:12:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462551 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2224.codfw.wmnet with OS bookworm... [15:16:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462589 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2226.codfw.wmnet with OS bookworm... [15:22:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462637 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2225.codfw.wmnet with OS bookworm... [15:30:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462729 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2192.codfw.wmnet with OS book... [15:30:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462733 (10ops-monitoring-bot) pool host wikikube-worker[2224-2227].codfw.wmnet by jelto@cumin1002 with reason: None [15:30:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10462734 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[2224-2227].codfw.wmn... [15:31:12] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T383764#10462745 (10Jelto) [15:41:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10462794 (10ops-monitoring-bot) depool host mw[1421-1425].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes [15:43:52] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10462823 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host mw[1421-142... [15:54:07] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10462901 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1421 to wikikube-worker1102... [16:00:33] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10462925 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1422 to wikikube-worker1103... [16:04:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10462950 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1423 to wikikube-worker1104... [16:09:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10462979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1424 to wikikube-worker1105... [16:15:26] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10463017 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2192.codfw.... [16:17:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463021 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1425 to wikikube-worker1106... [16:20:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10463029 (10ops-monitoring-bot) pool host wikikube-worker2192.codfw.wmnet by jelto@cumin1002 with reason: None [16:20:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10463031 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [16:20:52] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463033 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1102.e... [16:21:03] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1103.e... [16:21:09] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463036 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1104.e... [16:21:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463037 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1105.e... [16:21:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463040 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1106.e... [16:52:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463164 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1102.eqiad.wmnet with OS bookwor... [16:53:06] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463165 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1102.eqiad.wmnet with OS boo... [16:53:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463166 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1103.eqiad.wmnet with OS bookwor... [16:53:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463167 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1103.eqiad.wmnet with OS boo... [16:53:30] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463168 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1106.eqiad.wmnet with OS bookwor... [16:53:41] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463169 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1106.eqiad.wmnet with OS boo... [16:59:31] 06serviceops, 10wikitech.wikimedia.org: Wikitech displays desktop site on mobile devices - https://phabricator.wikimedia.org/T383656#10463197 (10jijiki) p:05Triage→03Low [16:59:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463199 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1104.eqiad.wmnet with OS bookwor... [17:02:08] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463210 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1103.eqiad.wmnet with OS bookwor... [17:02:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463211 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1105.eqiad.wmnet with OS bookwor... [17:02:22] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463212 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1106.eqiad.wmnet with OS bookwor... [17:02:46] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463213 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1102.eqiad.wmnet with OS bookwor... [17:05:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463226 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1105.eqiad.wmnet with OS boo... [17:05:30] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463227 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1106.eqiad.wmnet with OS boo... [17:05:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463228 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1102.eqiad.wmnet with OS boo... [17:06:32] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463233 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1103.eqiad.wmnet with OS boo... [17:43:25] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463575 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1106.eqiad.wmnet with OS bookwor... [17:47:35] 06serviceops: Performance assessment of PHP8.1 - https://phabricator.wikimedia.org/T383805 (10jijiki) 03NEW p:05Triage→03Low [17:47:46] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463623 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1105.eqiad.wmnet with OS bookwor... [17:49:17] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463644 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1102.eqiad.wmnet with OS bookwor... [17:52:30] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463677 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1103.eqiad.wmnet with OS bookwor... [17:53:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463685 (10ops-monitoring-bot) pool host wikikube-worker[1102-1106].eqiad.wmnet by kamila@cumin1002 with reason: None [17:53:18] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10463686 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1102-1106].eqiad.wm... [17:54:15] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T383620#10463696 (10kamila) [18:27:08] 06serviceops, 06Infrastructure-Foundations, 06Traffic: Ownership of the sre.deploy.hiddenparma cookbook - https://phabricator.wikimedia.org/T383809 (10Volans) 03NEW [18:41:36] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10463978 (10taavi) The `extensions/Translate/scripts/moveTranslatableBundle.php` script asks one to confirm a move by typing some text when prompted. This doesn't seem to work wit... [19:00:00] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T383764#10463999 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [21:35:47] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10464579 (10jijiki) [21:37:29] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10464582 (10jijiki) [23:23:59] 06serviceops, 10wikitech.wikimedia.org: Wikitech displays desktop site on mobile devices - https://phabricator.wikimedia.org/T383656#10464927 (10bd808) I'm not sure Wikitech has ever had a consistent mobile frontend experience. [23:40:56] 06serviceops: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10464953 (10Scott_French) With shellbox-syntaxhighlight now migrated and no issues encountered so far, now is a good time to figure out which services to migrate next. Again assuming that the main source of...