[08:31:27] 06serviceops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, and 2 others: Cross-datacenter Docker Registry replication broken since 2025-04-27 - https://phabricator.wikimedia.org/T413008#11480594 (10elukey) The sync seems now complete! Nice work :) Next steps before closing: update documentati... [08:36:55] 06serviceops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, and 2 others: Cross-datacenter Docker Registry replication broken since 2025-04-27 - https://phabricator.wikimedia.org/T413008#11480595 (10MatthewVernon) I think we're almost-but-not-quite-entirely caught up, but yes, I think the repor... [09:24:12] 06serviceops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, and 2 others: Cross-datacenter Docker Registry replication broken since 2025-04-27 - https://phabricator.wikimedia.org/T413008#11480607 (10elukey) 05Open→03Resolved a:03elukey Added https://wikitech.wikimedia.org/wiki/Docker-... [10:56:47] 06serviceops, 06collaboration-services, 07Developer Productivity, 13Patch-For-Review: deployment charts: automate testing on staging - https://phabricator.wikimedia.org/T413179#11480688 (10Clement_Goubert) >>! In T413179#11476581, @JMeybohm wrote: > FWIW there is the concept of helm test (https://helm.sh/d... [11:10:50] 06serviceops, 10RESTBase Sunsetting, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-24 (2025-12-02 to 2025-12-24)), 13Patch-For-Review: Reroute /api/rest_v1 documentation to REST Sandbox - https://phabricator.wikimedia.org/T396807#11480695 (10Clement_Goubert) CSP breaks JS and CSS on https://wikimedi... [12:00:29] 06serviceops, 10ConfirmEdit (CAPTCHA extension), 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): Increase in memcached get misses from mc1047 and mc1042 - https://phabricator.wikimedia.org/T412947#11480759 (10kostajh) > What options do we have here to avoid relying on cache misses? I'm not sure. I gue... [13:23:18] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker11XX - https://phabricator.wikimedia.org/T408749#11480841 (10Jclark-ctr) a:05Jhancock.wm→03Jclark-ctr [16:01:55] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker11XX - https://phabricator.wikimedia.org/T408749#11481183 (10Jclark-ctr) these are racked cabled and setup running into error with provisioning same as T407991 ` Retrieving the BMC's firmware version. BMC firmware release... [16:02:38] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481190 (10Jclark-ctr) [16:02:59] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481192 (10Jclark-ctr) [16:06:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481198 (10Clement_Goubert) Hmm it's very possible these hosts need UEFI and the partman recipe are wrong, on top of needing @elukey's [[ https://gerrit.wikimedia.org/r/c/... [16:10:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481210 (10Jclark-ctr) @Clement_Goubert thanks for checking these. I had not even looked at the patch yet. these would be UEFI [16:17:13] elukey: 1328 and up need UEFI partman which I totally forgot [16:17:21] elukey: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1220370 [16:19:31] wow just a few workers in eqiad :D [16:19:46] elukey: the partman thing is first match exit right? [16:19:51] elukey: yeah just a few... [16:20:47] yeah exactly, first one wins [18:01:02] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481625 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1328.eqiad.wmnet with OS trixie [18:22:27] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481731 (10Jclark-ctr) @elukey Ran into another provisioning issue. It looks like IPv4 PXE was disabled. The screenshot was taken after I changed it, and that was the only... [18:47:15] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481797 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1334.eqiad.wmnet with OS trixie [18:49:07] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481798 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1329.eqiad.wmnet with OS trixie [18:50:03] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481800 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1330.eqiad.wmnet with OS trixie [18:50:24] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481801 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1328.eqiad.wmnet with OS trixie completed: - wikiku... [18:50:54] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481802 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1331.eqiad.wmnet with OS trixie [18:52:30] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481803 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1332.eqiad.wmnet with OS trixie [18:52:33] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481804 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1333.eqiad.wmnet with OS trixie [19:20:26] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481842 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1334.eqiad.wmnet with OS trixie completed: - wikiku... [19:23:09] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481845 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1329.eqiad.wmnet with OS trixie completed: - wikiku... [19:30:55] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481870 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1332.eqiad.wmnet with OS trixie completed: - wikiku... [19:34:55] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481886 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1331.eqiad.wmnet with OS trixie completed: - wikiku... [19:39:54] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481900 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1330.eqiad.wmnet with OS trixie completed: - wikiku... [19:43:26] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481916 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1333.eqiad.wmnet with OS trixie completed: - wikiku... [19:47:09] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481922 (10Jclark-ctr) [19:48:28] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11481923 (10Jclark-ctr) 05Open→03Resolved [21:06:22] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482291 (10VRiley-WMF) wikikube-worker1360 B2 U18 CableID 5003 Port 27 wikikube-worker1361 B4 U36 CableID 5369 Port 47 wikikube-worker1362 C3 U37 CableID 230304500071 Por...