[07:15:31] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11130970 (10VRiley-WMF) [08:34:05] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11131072 (10VRiley-WMF) [08:52:30] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11131111 (10VRiley-WMF) [09:39:20] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11131205 (10VRiley-WMF) [09:56:35] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11131275 (10VRiley-WMF) [12:30:51] 06serviceops, 13Patch-For-Review: Provide nodejs22 base images for production - https://phabricator.wikimedia.org/T393437#11131699 (10MoritzMuehlenhoff) @Jdforrester-WMF: I've imported the new nodejs deb to the thirdparty/node22 and updated r1182837 to reflect the new apt source base base image. Could you also... [12:59:25] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11131766 (10Jclark-ctr) @Jhancock.wm These servers are still failing to image. @elukey @MoritzMuehlenhoff , could the older maps* entries in preseed.yaml be causing the issue? 'ma... [13:26:14] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11131825 (10MoritzMuehlenhoff) Can you please retry? I fixed a problem with the codfw install server setup earlier the (Euro) day, so it might simply work now. [14:11:57] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11131955 (10MoritzMuehlenhoff) >>! In T400637#11131765, @Jclark-ctr wrote: > @Jhancock.wm These servers are still failing to image. @elukey @MoritzMuehlenhoff , could the older maps*... [14:20:56] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11132084 (10Jhancock.wm) @Clement_Goubert i think their might be a mismatch regarding the site.pp on this one. I am not sure exactly what it is. we had a similar issue in T400195. Could... [14:23:15] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132086 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2011.codfw.wmnet with OS bookworm [14:38:24] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132140 (10Jclark-ctr) Mine would not even reach Debian installer [14:40:59] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132161 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host maps2012.codfw.wmnet with OS bookworm [14:43:08] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11132177 (10Clement_Goubert) >>! In T400485#11132084, @Jhancock.wm wrote: > @Clement_Goubert i think their might be a mismatch regarding the site.pp on this one. I... [14:46:51] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132219 (10Jclark-ctr) 2012 looks to be working now [15:21:43] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132365 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host maps2012.codfw.wmnet with OS bookworm completed: - maps2012 (**PASS**) - Rem... [15:28:03] 06serviceops, 13Patch-For-Review: Provide nodejs22 base images for production - https://phabricator.wikimedia.org/T393437#11132381 (10Jdforrester-WMF) >>! In T393437#11131699, @MoritzMuehlenhoff wrote: > @Jdforrester-WMF: I've imported the new nodejs deb to the thirdparty/node22 and updated r1182837 to reflect... [15:31:07] 06serviceops, 13Patch-For-Review: Provide nodejs22 base images for production - https://phabricator.wikimedia.org/T393437#11132393 (10MoritzMuehlenhoff) >>! In T393437#11132381, @Jdforrester-WMF wrote: >>>! In T393437#11131699, @MoritzMuehlenhoff wrote: >> @Jdforrester-WMF: I've imported the new nodejs deb to... [15:42:32] 06serviceops: Provide nodejs22 base images for production - https://phabricator.wikimedia.org/T393437#11132446 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Many thanks! [15:44:54] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132455 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host maps2013.codfw.wmnet with OS bookworm [15:45:10] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132457 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host maps2014.codfw.wmnet with OS bookworm [15:54:45] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132480 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host maps2011.codfw.wmnet with OS bookworm executed with errors: - maps2011 (**FA... [16:03:38] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132509 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host maps2011.codfw.wmnet with OS bookworm [16:26:39] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132541 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host maps2013.codfw.wmnet with OS bookworm completed: - maps2013 (**PASS**) - R... [16:31:30] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132558 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host maps2014.codfw.wmnet with OS bookworm completed: - maps2014 (**PASS**) - R... [16:38:31] 06serviceops: Prepare PHP 8.3 service images for Shellbox - https://phabricator.wikimedia.org/T403283 (10Scott_French) 03NEW [16:38:32] 06serviceops: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284 (10Scott_French) 03NEW [16:39:02] 06serviceops: Prepare PHP 8.3 service images for Shellbox - https://phabricator.wikimedia.org/T403283#11132597 (10Scott_French) 05Open→03In progress p:05Triage→03Medium [16:39:13] 06serviceops: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11132599 (10Scott_French) p:05Triage→03Medium [16:39:26] 06serviceops: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11132600 (10Scott_French) [16:39:27] 06serviceops: Prepare PHP 8.3 service images for Shellbox - https://phabricator.wikimedia.org/T403283#11132601 (10Scott_French) [16:45:39] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132640 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host maps2011.codfw.wmnet with OS bookworm completed: - maps2011 (**PASS**) - R... [16:46:52] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132643 (10Jhancock.wm) [16:47:57] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132646 (10Jhancock.wm) 05Open→03Resolved @MoritzMuehlenhoff looks like we got everything cleared up. all yours! [16:52:05] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11132654 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host deploy2003.codfw.wmnet with OS bookworm [16:52:27] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11132655 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host deploy2003.codfw.wmnet with OS bookworm executed with errors: - deploy2003 (**... [17:43:49] 06serviceops: Prepare PHP 8.3 service images for Shellbox - https://phabricator.wikimedia.org/T403283#11132883 (10Scott_French) 05In progress→03Resolved 8.3 image variants for all Shellbox services are now built and published in postmerge, and the test83 variant is run alongside test81 in the `shellbox-p... [17:55:14] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install maps201[1-4] - https://phabricator.wikimedia.org/T400637#11132908 (10MoritzMuehlenhoff) Thanks! [22:23:59] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install maps101[1-4] - https://phabricator.wikimedia.org/T400638#11133397 (10VRiley-WMF)