[08:49:19] 06serviceops, 10MW-on-K8s, 10Observability-Logging: Unexpected utilization increase in udp_localhost-info kafka-logging topic - https://phabricator.wikimedia.org/T384233#10478547 (10fgiunchedi) Thank you @Clement_Goubert ! I'm looking at the topic activity and expecting a similar increase in e.g. bytes or me... [09:46:58] 06serviceops, 10RESTBase Sunsetting, 10MW-Interfaces-Team (MWI-Sprint-1 (2025-01-14 to 2025-01-28)), 07User-notice: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10478698 (10akosiaris) >>! In T374683#10472491... [10:32:15] 06serviceops, 10MW-on-K8s, 10Observability-Logging: Unexpected utilization increase in udp_localhost-info kafka-logging topic - https://phabricator.wikimedia.org/T384233#10478790 (10jijiki) p:05Triage→03Medium [10:33:17] 06serviceops, 10MW-on-K8s, 10Observability-Logging: Unexpected utilization increase in udp_localhost-info kafka-logging topic - https://phabricator.wikimedia.org/T384233#10478791 (10jijiki) 05Open→03In progress [10:40:28] 06serviceops, 10MW-on-K8s, 10Observability-Logging: Unexpected utilization increase in udp_localhost-info kafka-logging topic - https://phabricator.wikimedia.org/T384233#10478810 (10fgiunchedi) 05In progress→03Stalled Wiggling the `retention.bytes` config for `udp_localhost-info` (as per https://wikitech... [10:47:32] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Staging error: Error from mapnik internals when requesting a snapshot with an overlay - https://phabricator.wikimedia.org/T384285 (10Jgiannelos) 03NEW [10:48:02] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Staging error: Error from mapnik internals when requesting a snapshot with an overlay - https://phabricator.wikimedia.org/T384285#10478863 (10Jgiannelos) [12:02:50] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294 (10jijiki) 03NEW [12:08:29] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10479188 (10jijiki) [12:47:07] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10479258 (10jijiki) [14:05:25] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Foundational Technology Requests, 10Wikidata, 10Wikidata-Query-Service: API Gateway to provide authorization and capacity management for W[CD]QS - https://phabricator.wikimedia.org/T313813#10479639 (10AntiCompositeNumber) 05In progress→03Open [15:18:40] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Move kafka-main2010 within the same rack - https://phabricator.wikimedia.org/T381788#10480097 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ffdb0a96-3214-40cf-acd0-ec05d4bf5539) set by jayme@cumin1002 for 2:00:00 on 1 host(s) and their servi... [15:40:34] 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, 06Traffic: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10480245 (10hnowlan) https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/973362 has been merged (wrong ticket tagged in... [15:55:34] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480322 (10ops-monitoring-bot) depool host mw[1470-1475].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes [16:03:44] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Move kafka-main2010 within the same rack - https://phabricator.wikimedia.org/T381788#10480348 (10JMeybohm) >>! In T381788#10480097, @ops-monitoring-bot wrote: > Icinga downtime and Alertmanager silence (ID=ffdb0a96-3214-40cf-acd0-ec05d4bf5539) set by jayme@cumin1... [16:04:14] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480349 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host mw[1470-1475].eqiad.wmnet com... [16:12:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480389 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1470 to wikikube-worker1123 completed: - mw147... [16:16:18] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480402 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1471 to wikikube-worker1124 completed: - mw147... [16:17:01] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team, 13Patch-For-Review: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10480405 (10Joe) @jijiki I've looked into JIT in php 8.1 a bit, and it seems like it had several serious bugs (see for exam... [16:20:54] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480418 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1472 to wikikube-worker1125 completed: - mw147... [16:25:30] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480428 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1473 to wikikube-worker1126 completed: - mw147... [16:30:35] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480442 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1474 to wikikube-worker1127 completed: - mw147... [16:34:58] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480470 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from mw1475 to wikikube-worker1128 completed: - mw147... [16:44:13] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480503 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1123.eqiad.wmnet with OS... [16:44:16] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480504 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1124.eqiad.wmnet with OS... [16:44:29] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480505 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1125.eqiad.wmnet with OS... [16:44:34] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480506 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1126.eqiad.wmnet with OS... [16:44:56] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480509 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1127.eqiad.wmnet with OS... [16:45:11] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480512 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1128.eqiad.wmnet with OS... [17:06:43] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Move kafka-main2010 within the same rack - https://phabricator.wikimedia.org/T381788#10480665 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm This has been completed. Thank you for your help! [17:24:21] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480792 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1123.eqiad.wmnet with OS boo... [17:26:18] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-codfw, 06SRE: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10480800 (10Jelto) >>! In T383709#10480784, @Papaul wrote: > @Jelto when do you think will be a best time for you or so... [17:28:28] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480810 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1124.eqiad.wmnet with OS boo... [17:30:34] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-codfw, 06SRE: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10480819 (10Papaul) @Jelto thanks please let us know when best works for you for the gerrit2002. Thanks [17:32:15] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480827 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1125.eqiad.wmnet with OS boo... [17:35:44] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480841 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1128.eqiad.wmnet with OS boo... [17:38:18] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480847 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1127.eqiad.wmnet with OS boo... [17:42:14] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480850 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1126.eqiad.wmnet with OS boo... [17:43:14] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-codfw, 06SRE: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10480855 (10JMeybohm) mw2259 and mw2278 are to be decommed (T354791, T384043) mw2355 is now wikikube-worker2229 (T383862... [17:58:44] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-codfw, 06SRE: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10480893 (10Jhancock.wm) [18:00:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10480908 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1126.eqiad.wmnet with OS... [18:39:01] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10481217 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1126.eqiad.wmnet with OS boo... [18:40:56] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T383620#10481232 (10kamila) [18:53:40] 06serviceops, 13Patch-For-Review: Support cookie-driven fractional migration to PHP 8.1 deployments of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T377042#10481304 (10Scott_French) Alas, the first couple of logs were missed while stashbot was absent from -operations: ` 18:02:44 !l... [18:57:40] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06MediaWiki-Platform-Team, 13Patch-For-Review: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10481330 (10Ottomata) [18:57:53] 06serviceops, 06MediaWiki-Platform-Team, 13Patch-For-Review: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10481332 (10Ottomata) [19:04:12] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1-flavored k8s deployments for all MediaWiki services - https://phabricator.wikimedia.org/T377040#10481357 (10Scott_French) 05Open→03Resolved Many thanks to @jijiki for merging those last two patches. Next steps will come in T383845 (pre-ramp capacity au... [20:38:58] 06serviceops, 06MediaWiki-Platform-Team, 13Patch-For-Review: Add support for JIT in PHP8.1 images - https://phabricator.wikimedia.org/T384294#10481694 (10Scott_French) Ah, interesting! I was unaware of the changes in 8.3 mentioned in T384294#10480405. If that's the case, then I wonder whether it makes sense... [22:31:26] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10482179 (10Scott_French) Alright, I've uploaded patches to move the -video and -constraints variants forward next. Points of note: **shellbox-constraints**: This uses a similar (thoug...