[00:01:49] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11519531 (10VRiley-WMF) [00:04:20] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11519533 (10VRiley-WMF) [00:09:07] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11519535 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host wikikube-worker1370.eqiad.wmnet with OS trixie [00:14:33] 06serviceops, 06Data-Persistence, 13Patch-For-Review: Deploy instance of hoarde as artifact-cache(?) in k8s - https://phabricator.wikimedia.org/T414112#11519537 (10Eevans) >>! In T414112#11519172, @Ottomata wrote: > [ ... ] > > Hm, what about using the term 'foreign' as in foreign key somehow. //foreign-ar... [00:15:39] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11519541 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host wikikube-worker1372.eqiad.wmnet with OS trixie completed: - wikikub... [00:41:10] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11519577 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host wikikube-worker1370.eqiad.wmnet with OS trixie completed: - wikikub... [00:41:21] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11519578 (10VRiley-WMF) 05Open→03Resolved This has been completed [09:52:11] 06serviceops, 10ConfirmEdit (CAPTCHA extension), 06Product Safety and Integrity, 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): Increase in memcached get misses from mc1047 and mc1042 - https://phabricator.wikimedia.org/T412947#11520342 (10kostajh) [10:09:59] 06serviceops, 10RESTBase Sunsetting, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-25 (2026-01-13 to 2026-01-27)), 13Patch-For-Review: Reroute /api/rest_v1 documentation to REST Sandbox - https://phabricator.wikimedia.org/T396807#11520372 (10Clement_Goubert) ATS patch redeployed, it looks ok to me,... [10:20:22] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install wikikube-worker2332-56 - https://phabricator.wikimedia.org/T408757#11520422 (10Clement_Goubert) >>! In T408757#11513905, @Jhancock.wm wrote: > @Clement_Goubert the servers landed last week. Gonna start unpacking them tomorrow or wednesday. A... [11:06:50] 06serviceops, 13Patch-For-Review: Move EXCLUDED_SERVICES attribute from sre.discovery.datacentre to service catalog - https://phabricator.wikimedia.org/T412211#11520571 (10Blake) [11:11:35] 06serviceops, 13Patch-For-Review: Move EXCLUDED_SERVICES attribute from sre.discovery.datacentre to service catalog - https://phabricator.wikimedia.org/T412211#11520598 (10Blake) The old constant has been removed, so the last task here will be to update https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Spe... [12:17:23] 06serviceops: Move EXCLUDED_SERVICES attribute from sre.discovery.datacentre to service catalog - https://phabricator.wikimedia.org/T412211#11520746 (10Blake) The documentation has been updated; nothing left to do. [12:17:29] 06serviceops: Move EXCLUDED_SERVICES attribute from sre.discovery.datacentre to service catalog - https://phabricator.wikimedia.org/T412211#11520747 (10Blake) 05Open→03Resolved [15:11:58] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11521553 (10elukey) It happens with all the image push tests, with Docker on bullseye and bookworm (tried both build nodes). I dumped the registry's goroutines when... [15:39:26] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11521716 (10elukey) Meanwhile this is the stacktrace for dockerd and the relevant goroutine: ` goroutine 148 [select, 3 minutes]: net/http.(*persistConn).roundTrip... [16:36:11] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#11522110 (10elukey) As the last times, I went afk for ~1h, got back and retried the same docker push that completed immediately without hanging. [23:14:32] 06serviceops: Document where the docker images used by WMF are located - https://phabricator.wikimedia.org/T412787#11523460 (10Raine) p:05Triage→03Low