[09:14:18] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11532568 (10elukey) After adding the envoy access logs (they do log only HTTP 500+ requests though): ` [2026-01-19T09:04:41.009Z] "PUT /registry-restricted/docker/... [09:56:01] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11532932 (10elukey) I think this is probably related to some weird state the the bucket is in: ` elukey@stat1010:~$ s3cmd del s3://registry-restricted/docker/regis... [10:01:56] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11532993 (10elukey) Finally something that makes sense - on stat1010 I tried to upload a super small fine (a txt file with a date) and this is the result: ` elukey... [10:38:20] 06serviceops, 10Ceph, 06Data-Persistence, 06SRE, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11533132 (10elukey) [11:02:35] 06serviceops, 10Ceph, 06Data-Persistence, 06SRE, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11533214 (10MatthewVernon) After the deletion of objects from registry-restricted (from both eqiad and codfw) late last week, we were stuck with sync being... [11:04:18] 06serviceops, 10Ceph, 06Data-Persistence, 06SRE, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11533218 (10MatthewVernon) [if that report is wrong, then probably a full re-sync is required :-/ ] [11:26:48] 06serviceops, 06Infrastructure-Foundations, 06Traffic: Ownership of the sre.deploy.hiddenparma cookbook - https://phabricator.wikimedia.org/T383809#11533270 (10MLechvien-WMF) AFAICT this does not fall in the scope of cookbooks Serviceops maintains so removing that tag, but please reach out to me if any doubts. [11:36:14] 06serviceops, 10Ceph, 06Data-Persistence, 06SRE, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11533333 (10elukey) After the above maintenance I don't see any docker or s3cmd push problem, so all this was apparently due to the ceph's replication. [12:16:35] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission wikikube-worker[2003-2004,2007-2010,2019-2032,2040,2043,2045,2048].codfw.wmnet - https://phabricator.wikimedia.org/T409102#11533457 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by cgoubert@cumin1003 for... [13:03:30] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission wikikube-worker[2003-2004,2007-2010,2019-2032,2040,2043,2045,2048].codfw.wmnet - https://phabricator.wikimedia.org/T409102#11533586 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by cgoubert@cumin1003 for... [13:35:46] 06serviceops, 06Commons, 10MediaWiki-Uploading: Add metrics for Commons file uploads - https://phabricator.wikimedia.org/T385707#11533757 (10hnowlan) [14:49:30] 06serviceops, 10Ceph, 06Data-Persistence, 06SRE, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11534166 (10elukey) Next steps: 1) Clean up the bucket from all the tests via s3cmd (only from a DC) and check replication. 2) Try to push and pull an ima... [15:28:39] 06serviceops, 06MW-Interfaces-Team, 06Traffic, 07Epic, and 3 others: Epic: API Rate Limiting Architecture - https://phabricator.wikimedia.org/T399291#11534337 (10matmarex) [15:30:15] 06serviceops, 06MediaWiki-Platform-Team, 06MW-Interfaces-Team, 07Epic, 07OKR-Work: API tokens: use rate limit classes instead of rate limit overrides. - https://phabricator.wikimedia.org/T409305#11534339 (10matmarex) [15:30:37] 06serviceops, 06MediaWiki-Platform-Team, 06MW-Interfaces-Team, 07Epic, 07OKR-Work: API tokens: use rate limit classes instead of rate limit overrides. - https://phabricator.wikimedia.org/T409305#11534340 (10matmarex) a:03pmiazga [16:04:06] 06serviceops, 10Ceph, 06Data-Persistence, 06SRE, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11534500 (10elukey) I was able to clean up the whole bucket with recursive calls in few minutes, meanwhile the other day I frequently got HTTP 504s. So poi... [16:08:22] 06serviceops, 10Ceph, 06Data-Persistence, 06SRE, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11534522 (10elukey) Tried to push and pull one image, super fast: ` elukey@build2002:~$ sudo docker push registry1004.eqiad.wmnet:5002/calico/typha Using... [16:33:31] 06serviceops, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07Wikimedia-Performance-recommendation: Add support for JIT in PHP 8.4 images - https://phabricator.wikimedia.org/T384294#11534669 (10MLechvien-WMF) a:05jijiki→03Scott_French Hi Scott, do you know if this task still makes sense after... [16:57:29] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: sre.discovery.datacenter should handle depooled authdns hosts - https://phabricator.wikimedia.org/T375285#11534757 (10MLechvien-WMF) @Blake @Scott_French could you assess what part of this is still valid (please edit the description) and see what we...