[06:11:48] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482775 (10VRiley-WMF) [06:26:24] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482785 (10VRiley-WMF) [06:34:24] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11482787 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host wikikube-worker1360.eqiad.wmnet with OS trixie [06:49:23] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482825 (10VRiley-WMF) [07:06:55] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11482833 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host wikikube-worker1360.eqiad.wmnet with OS trixie completed: - wi... [07:32:31] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482844 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host wikikube-worker1362.eqiad.wmnet with OS trixie [07:50:28] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482846 (10VRiley-WMF) [08:06:00] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482849 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host wikikube-worker1362.eqiad.wmnet with OS trixie completed: - wikikub... [08:06:38] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482850 (10VRiley-WMF) [08:17:33] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482863 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host wikikube-worker1363.eqiad.wmnet with OS trixie [09:37:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482926 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host wikikube-worker1363.eqiad.wmnet with OS trixie executed with errors... [10:37:34] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11482977 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host wikikube-worker1363.eqiad.wmnet with OS trixie [10:42:19] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11482980 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host wikikube-worker1361.eqiad.wmnet with OS trixie [10:57:58] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11482989 (10elukey) 05Resolved→03Open >>! In T408749#11481731, @Jclark-ctr wrote: > @elukey Ran into another provisioning issue. It looks like IPv4 PXE was disabled. Th... [10:59:51] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11482994 (10Jclark-ctr) It was all of them failed to image over http but when I enabled ode they imagined [11:00:37] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11483006 (10VRiley-WMF) [11:16:17] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11483024 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host wikikube-worker1361.eqiad.wmnet with OS trixie completed: - wikiku... [11:57:52] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11483077 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host wikikube-worker1363.eqiad.wmnet with OS trixie executed with errors... [13:21:13] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Add configuration for MESH_CHECK_SKIP in periodic job puppet definition - https://phabricator.wikimedia.org/T412818#11483144 (10Drvincent1995) a:03Drvincent1995 I am an Herbalist & Spell caster ,Do you need any spiritual help in any kind of situation: Love s... [13:41:21] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Add configuration for MESH_CHECK_SKIP in periodic job puppet definition - https://phabricator.wikimedia.org/T412818#11483173 (10Clement_Goubert) a:05Drvincent1995→03Clement_Goubert [13:53:37] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11483201 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host wikikube-worker1364.eqiad.wmnet with OS trixie [14:09:02] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11483248 (10VRiley-WMF) [14:23:49] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11483271 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host wikikube-worker1364.eqiad.wmnet with OS trixie completed: - wikikub... [14:29:56] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11483278 (10elukey) @Jclark-ctr got it, do you remember more precisely the error while doing HTTP boot? Did you see any failure registered when the host tried to do it etc..? [14:39:50] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11483302 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1328.eqiad.wmnet with OS trixie [14:43:59] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11483304 (10Jclark-ctr) @elukey Although it was not working yesterday, it seems to be working now. I will revert all the servers back to PXE off and reimage them to verify. [14:49:55] 06serviceops, 03[MWI] FY2025-26 Q3, 07Epic, 06MW-Interfaces-Team (MWI-Roadmap), 07OKR-Work: [Hypothesis] 5.2.2c: Reroute API Portal Endpoints - https://phabricator.wikimedia.org/T413438 (10HCoplin-WMF) 03NEW [15:15:11] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11483371 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1328.eqiad.wmnet with OS trixie completed: - wikiku... [15:19:14] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker13[28-34] - https://phabricator.wikimedia.org/T408749#11483395 (10Jclark-ctr) 05Open→03Resolved [19:10:07] 06serviceops, 06MW-Interfaces-Team, 07Epic, 07OKR-Work: [5.2.2c Epic]: API Gateway Rerouting Testing - https://phabricator.wikimedia.org/T413449 (10HCoplin-WMF) 03NEW [19:11:16] 06serviceops, 03[MWI] FY2025-26 Q3, 07Epic, 07OKR-Work: [5.2.2c Epic]: API Gateway Rerouting Testing - https://phabricator.wikimedia.org/T413449#11483719 (10HCoplin-WMF)