[08:07:10] 06serviceops, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), 10MW-Interfaces-Team (MWI-Sprint-1 (2025-01-14 to 2025-01-28)), 07User-notice: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title end... - https://phabricator.wikimedia.org/T374683#10469299 [08:33:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469307 (10ops-monitoring-bot) depool host mw[2282,2310-2311].codfw.wmnet by jelto@cumin1002 with reason: Renaming nodes [08:35:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469309 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host mw[2282,2310... [09:01:57] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469323 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2310 to wikikube-worker2240 c... [09:06:40] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 10Data-Platform-SRE (2025.01.11 - 2025.01.31), 07Kubernetes: Remove the kubelet readOnlyPort - https://phabricator.wikimedia.org/T383413#10469335 (10Gehel) [09:08:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469338 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2311 to wikikube-worker2241 c... [09:20:10] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: decommission mw2282.codfw.wmnet - https://phabricator.wikimedia.org/T383965 (10Jelto) 03NEW [09:33:28] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469408 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2240.co... [09:39:05] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469420 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2241.co... [09:55:51] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T383862#10469458 (10Jelto) [10:08:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469484 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2241.codfw.... [10:08:43] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469485 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2241.co... [10:19:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469511 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2240.codfw.... [10:51:53] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469627 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2241.codfw.... [10:58:15] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469647 (10ops-monitoring-bot) pool host wikikube-worker[2240-2241].codfw.wmnet by jelto@cumin1002 with reason: None [10:58:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469648 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [11:01:45] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10469657 (10Jelto) 05Open→03Resolved a:03Jelto All wikikube-worker in codfw are on bookworm, containerd and in the new VL... [11:09:41] 06serviceops, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), 10MW-Interfaces-Team (MWI-Sprint-1 (2025-01-14 to 2025-01-28)), and 2 others: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoi... - https://phabricator.wikimedia.org/T374683#10469697 [11:31:46] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10469755 (10ops-monitoring-bot) pool host wikikube-worker[1111-1116].eqiad.wmnet by kamila@cumin1002 with reason: None [12:07:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10469889 (10ops-monitoring-bot) pool host wikikube-worker[1111-1116].eqiad.wmnet by kamila@cumin1002 with reason: None [12:07:38] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10469890 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1111-1116].eqiad.wm... [12:56:24] 06serviceops, 10MW-on-K8s, 10Observability-Metrics: Move mw-accesslog-metrics instance to k8s - https://phabricator.wikimedia.org/T371214#10470293 (10kamila) 05Open→03Stalled I did not work on this because I am not sure about running it in wikikube. Since people are making progress on k8s-aux in codfw, I... [12:58:06] 06serviceops: operations/docker-images/production-images contains references to non-existent image python3 - https://phabricator.wikimedia.org/T336682#10470306 (10elukey) @hashar as stated in T336682#10014067 plus a look in codesearch there is nothing that really binds to the old Python3 images. If anything come... [13:10:13] 06serviceops, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), 10MW-Interfaces-Team (MWI-Sprint-1 (2025-01-14 to 2025-01-28)), and 2 others: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoi... - https://phabricator.wikimedia.org/T374683#10470345 [13:28:57] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10470447 (10Jgiannelos) It gets more complicated. Even with outgoing connections allowed (in my local env) geoshapes fails to render ove... [14:33:09] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next - https://phabricator.wikimedia.org/T379986#10470871 (10Jdforrester-WMF) @Krinkle: T380485 and T383633 are marked as blockers of this and... [15:26:54] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Snapshots with overlay map failed to render - https://phabricator.wikimedia.org/T384023 (10Jgiannelos) 03NEW [15:27:19] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Snapshots with overlay map failed to render - https://phabricator.wikimedia.org/T384023#10471217 (10Jgiannelos) I tested this patch locally and renders the map with overlay as expected from prod: https://github.com/wikimedia/geojso... [15:45:27] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next - https://phabricator.wikimedia.org/T379986#10471393 (10Scott_French) Given the scope of this task (functional testing via mwdebug-next),... [15:45:49] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485#10471394 (10Scott_French) [15:45:53] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next - https://phabricator.wikimedia.org/T379986#10471395 (10Scott_French) [15:46:06] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10471396 (10Scott_French) [15:46:22] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485#10471402 (10Scott_French) Updating parent task to T319432, as this does not block T379986. [16:19:52] 06serviceops, 10Continuous-Integration-Infrastructure, 10MediaWiki-Platform-Team (Radar): Prepare WMF PHP 8.1 packages for Bullseye - https://phabricator.wikimedia.org/T372507#10471502 (10Scott_French) [16:19:55] 06serviceops, 10Continuous-Integration-Infrastructure: Prepare PHP 8.1 production images - https://phabricator.wikimedia.org/T372602#10471503 (10Scott_French) [16:53:12] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Improve calico-typha firewall rules - https://phabricator.wikimedia.org/T365687#10471662 (10JMeybohm) a:05jijiki→03JMeybohm I've configured staging-codfw to run with mTLS enabled. Certificates have a 72h expiry there, so we should be ab... [16:55:41] 06serviceops: Create Grafana and Logstash dashboards for MediaWiki migrations/upgrades - https://phabricator.wikimedia.org/T383875#10471680 (10Scott_French) An additional point of note on logstash: The mediawiki logs emitted by the -next and -migration releases can be easily distinguished by their `servergroup`... [17:05:16] 06serviceops, 10MW-on-K8s, 06SRE, 13Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#10471771 (10hnowlan) [17:10:32] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian), 13Patch-For-Review: Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10471819 (10Jgiannelos) Other than the patch above we have another reference of kartotherian issuing outgoing requ... [17:15:42] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian), 13Patch-For-Review: Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10471880 (10Jgiannelos) [17:16:13] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Staging error: Snapshots with overlay map failed to render - https://phabricator.wikimedia.org/T384023#10471893 (10Jgiannelos) [17:16:20] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Staging error: Snapshots with overlay map failed to render - https://phabricator.wikimedia.org/T384023#10471894 (10Jgiannelos) a:03Jgiannelos [17:16:56] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian), 13Patch-For-Review: Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10471897 (10Jgiannelos) a:03Jgiannelos [17:29:36] 06serviceops, 06SRE, 06SRE-OnFire, 10Release-Engineering-Team (Radar), 07Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162#10471941 (10LSobanski) #serviceops own the deployment servers, reassigning. [17:59:18] 06serviceops, 10MW-on-K8s, 06SRE, 13Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#10472018 (10hnowlan)