[06:23:44] Hi! I'm planning to deploy https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikistories/+/1134400 in a bit. Any concerns? [06:23:53] I never know which channel to ask in.... [07:48:11] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10716525 (10elukey) We may have a different version of the problem now, after the docker push serialization. During today's early EU morning deployment, scap got stuck for ages in the image push step.... [08:05:48] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10SRE-tools: Create a cookbook to automate gerrit's switchover - https://phabricator.wikimedia.org/T260666#10716569 (10ABran-WMF) [09:40:35] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10716962 (10elukey) The only occurrence of this issue happens with the multiversion image, so there must be something in it that sometimes hits a limit in the registry. I tried the following: ` eluk... [10:40:16] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10717103 (10elukey) I'd like to start checking the swift logs, here some useful data: * grep in this file: /var/log/swift/proxy-access.log * target: swift-proxy-codfw * swift access log format: https:... [13:11:30] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission kafka-main1001 / kafka-main1002 / kafka-main1003 / kafka-main1004 / kafka-main1005 - https://phabricator.wikimedia.org/T381593#10717634 (10jijiki) a:05jijiki→03VRiley-WMF [13:12:20] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission kafka-main1001 / kafka-main1002 / kafka-main1003 / kafka-main1004 / kafka-main1005 - https://phabricator.wikimedia.org/T381593#10717639 (10jijiki) @VRiley-WMF sorry this fell through the cracks, kafka-main1005 has been dec... [13:39:49] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission kafka-main1001 / kafka-main1002 / kafka-main1003 / kafka-main1004 / kafka-main1005 - https://phabricator.wikimedia.org/T381593#10717766 (10VRiley-WMF) [13:40:38] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission kafka-main1001 / kafka-main1002 / kafka-main1003 / kafka-main1004 / kafka-main1005 - https://phabricator.wikimedia.org/T381593#10717771 (10VRiley-WMF) 05Open→03Resolved [13:56:01] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10717834 (10elukey) I tried to run something like the following, concentrating on the second deployment and the layer causing errors, `e4ee3a8b399ef2645222ef14ea3ddfb179d5f2da58bf38424713d267dbaf6acb`... [14:33:27] 06serviceops, 06Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 2 others: Wikifeeds increase on 500 errors after switchover to core page HTML - https://phabricator.wikimedia.org/T354657#10717993 (10Jgiannelos) 05Open→03Resolved a:03Jgiannelos Closing this one since there are no... [14:36:24] 06serviceops, 10API Platform (RESTBase Deprecation Roadmap): Remove recommendation-api from the REST API offerings - https://phabricator.wikimedia.org/T390517#10718041 (10akosiaris) [15:08:52] 06serviceops, 06MW-Interfaces-Team: Migrate mw-interfaces-team jobs to mw-cron - https://phabricator.wikimedia.org/T388541#10718190 (10HCoplin-WMF) p:05Triage→03Medium [16:18:28] 06serviceops, 10Shellbox, 10SyntaxHighlight, 10MW-1.44-notes (1.44.0-wmf.24; 2025-04-08), 07Wikimedia-production-error: Shellbox bubbles GuzzleHttp\Exception\ConnectException when it should probably wrap it in a ShellboxError? - https://phabricator.wikimedia.org/T374117#10718527 (10Jdforrester-WMF) ... [17:42:43] 06serviceops: Turn down MediaWiki image builds for PHP 7.4 - https://phabricator.wikimedia.org/T391057#10718774 (10Scott_French) [17:42:48] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10718775 (10Scott_French) [18:56:47] 06serviceops, 06Discovery-Search, 07Analytics-Data-Problem, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Search Update Pipeline requests to Action API are logged as coming from 127.0.0.1 - https://phabricator.wikimedia.org/T388855#10719043 (10nshahquinn-wmf) [18:56:57] 06serviceops, 06Release-Engineering-Team, 10Scap, 06SRE-OnFire, 10Sustainability (Incident Followup): Should scap be able to update helmfile-defaults when -Dbuild_mw_container_image:False ? - https://phabricator.wikimedia.org/T390531#10719044 (10dancy) [20:26:34] 06serviceops, 10Scap: Migrate scap's maintenance script invocations to PHP 8.1 - https://phabricator.wikimedia.org/T390225#10719278 (10Scott_French) I was able to do some very testing in train-dev by switching `mediawiki_runtime_image` to `php8.1-fpm-multiversion-base` and deploying with scap (after clearing o...