[07:07:51] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 10Content-Transform-Team (Work In Progress): PCS caching metrics: Add label to distinct production vs pregeneration traffic - https://phabricator.wikimedia.org/T392052 (10Jgiannelos) 03NEW [07:21:57] 06serviceops, 13Patch-For-Review: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10746431 (10elukey) A couple of ideas: 1) There is also another thing to keep into consideration, this is the first time that we observe this issue. We have other Docker images c... [10:20:55] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🧭 Northward Datacentre Switchover (March 2025) - https://phabricator.wikimedia.org/T385155#10746795 (10hnowlan) 05Open→03Resolved [10:21:11] 06serviceops, 07Datacenter-Switchover: SRE comms for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T385157#10746796 (10hnowlan) 05Open→03Resolved [10:49:03] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10746916 (10Clement_Goubert) ` mwscript-k8s --php_version 8.1 -f -- ./extensions/GrowthExperiments/maintenance/updateMenteeD... [11:08:24] 06serviceops, 10Continuous-Integration-Config, 13Patch-For-Review: Several recent slow (>15 minute) helm-lint job runs - https://phabricator.wikimedia.org/T387781#10747006 (10kamila) [12:00:50] 06serviceops, 10Wikimedia-Site-requests, 07Performance Issue: Choose a sensible set of thumbnail sizes for Special:Preferences - https://phabricator.wikimedia.org/T106640#10747187 (10Ladsgroup) FWIW, it's not as easy as it looks since mediawiki stores the preferences as array keys so if we remove one size, e... [12:19:21] 06serviceops, 10CampaignEvents, 06Campaigns-Product-Team, 10MW-on-K8s, 13Patch-For-Review: Migrate CampaignEvents jobs to mw-cron - https://phabricator.wikimedia.org/T385867#10747245 (10Clement_Goubert) p:05Triage→03Medium [12:25:26] 06serviceops, 10CampaignEvents, 06Campaigns-Product-Team, 10MW-on-K8s, 13Patch-For-Review: Migrate CampaignEvents jobs to mw-cron - https://phabricator.wikimedia.org/T385867#10747259 (10Clement_Goubert) [12:53:57] 06serviceops, 13Patch-For-Review: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10747420 (10akosiaris) Couple of notes as well * scap is the only use case where multiple images are built and pushed together in short timeframes and also tried to be used immed... [13:15:52] 06serviceops, 10Continuous-Integration-Config, 13Patch-For-Review: Several recent slow (>15 minute) helm-lint job runs - https://phabricator.wikimedia.org/T387781#10747502 (10Joe) >>! In T387781#10737662, @hashar wrote: > I think that should be changed to compare against the parent instead (`HEAD^`). As exp... [15:37:46] 06serviceops, 06Abstract Wikipedia team, 10function-orchestrator: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10748477 (10Jdforrester-WMF) [15:38:05] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10748479 (10Jdforrester-WMF) [16:24:20] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10748775 (10DSantamaria) p:05Triage→03High [16:35:12] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10748818 (10Scott_French) 05In progress→03Resolved Now that 7.4-based images are no longer being built and the 'migration' releases have been normalized for their next use (8.3)... [16:57:52] 06serviceops, 06Release-Engineering-Team: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10748926 (10dduvall) →14Duplicate dup:03T390251 [16:58:01] 06serviceops, 13Patch-For-Review: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10748928 (10dduvall) [16:58:14] 06serviceops, 06Release-Engineering-Team: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10748931 (10dduvall) Closed as a duplicate that, while ongoing, is not strictly a train blocker. If folks run into this issue during a deployment, they should... [17:06:50] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move mwscript wrapper from base image to copy on build - https://phabricator.wikimedia.org/T391665#10748973 (10Clement_Goubert) 05In progress→03Resolved [20:49:26] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: relocate (3) service-ops hosts out of eqiad D6 - https://phabricator.wikimedia.org/T391599#10750017 (10Eevans) > sessionstore1006: > [] (service owner) Does the host need to stay in row D and keep its IP/VLAN? It //does// need to stay in row D, yes. If the IP/V... [20:52:11] 06serviceops, 10Abstract Wikipedia team (25Q4 (Apr–Jun)): Provide guidance on how to use apache bench to benchmark requests not through SSL for production services - https://phabricator.wikimedia.org/T390099#10750028 (10ecarg) 05In progress→03Resolved a:03ecarg TY all for the advice! Marking as 'Reso... [22:00:34] 06serviceops, 10Wikimedia-Site-requests, 07Performance Issue: Choose a sensible set of thumbnail sizes for Special:Preferences - https://phabricator.wikimedia.org/T106640#10750311 (10Reedy) [22:16:30] 06serviceops, 10Wikimedia-Site-requests, 07Performance Issue: Choose a sensible set of thumbnail sizes for Special:Preferences - https://phabricator.wikimedia.org/T106640#10750371 (10matmarex) >>! In T106640#10747187, @Ladsgroup wrote: > FWIW, it's not as easy as it looks since mediawiki stores the preferenc... [22:19:02] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: relocate (3) service-ops hosts out of eqiad D6 - https://phabricator.wikimedia.org/T391599#10750374 (10RobH) 05Open→03Stalled a:05Kappakayala→03RobH Please note this needs to be stalled as it turns out we may not use D6 for frack. Please take no further...