[00:27:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T394333) [00:27:20] T394333: Q4:rack/setup/install cloudcephosd10[48-51] & relocate cloudcephosd1039 - https://phabricator.wikimedia.org/T394333 [08:38:49] (03CR) 10Majavah: [C:03+2] Add PHAB contact group member type [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1151617 (https://phabricator.wikimedia.org/T394446) (owner: 10Majavah) [08:39:03] (03CR) 10Majavah: [C:03+2] alertmanager: Support PHAB contact group members [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1151618 (https://phabricator.wikimedia.org/T394446) (owner: 10Majavah) [08:39:16] (03Merged) 10jenkins-bot: Add PHAB contact group member type [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1151617 (https://phabricator.wikimedia.org/T394446) (owner: 10Majavah) [08:39:40] (03Merged) 10jenkins-bot: alertmanager: Support PHAB contact group members [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1151618 (https://phabricator.wikimedia.org/T394446) (owner: 10Majavah) [08:39:55] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Consider setting up an https://github.com/knyar/phalerts instance in metricsinfra - https://phabricator.wikimedia.org/T394446#10874423 (10taavi) @bd808 I'm guessing you're after a rule that sends all deployment-prep aler... [09:32:39] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Consider setting up an https://github.com/knyar/phalerts instance in metricsinfra - https://phabricator.wikimedia.org/T394446#10874619 (10taavi) [09:36:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-controller-2 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:40:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [09:51:28] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-controller-2 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:55:28] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [10:29:40] 10Quarry: quarry is leaking tmp files - https://phabricator.wikimedia.org/T395237#10874757 (10taavi) 05Resolved→03Open a:05taavi→03None Still happening. [10:30:22] 14cloud-services-team (Kanban), 06DBA, 07Puppet: labtestpuppetmaster2001 is failing to backup - https://phabricator.wikimedia.org/T256846#10874763 (10jcrespo) Ignore my suggestion, sorry, it was done already. Current status: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/9ae1d81b4ae21... [12:08:28] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10875153 (10Andrew) @Jhancock.wm I updated the preseed rule for this server and it should make a SW raid now. If it still fails you ca... [12:09:11] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cloudcephosd10[48-51] & relocate cloudcephosd1039 - https://phabricator.wikimedia.org/T394333#10875158 (10Andrew) a:05Andrew→03None Site.pp is updated and cloudcephosd1039 is drained and ready to... [12:14:48] 06cloud-services-team, 10Cloud-VPS: Add monitoring for Metadata service - https://phabricator.wikimedia.org/T395797 (10taavi) 03NEW p:05Triage→03Medium [12:25:16] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cloudcephosd10[48-51] & relocate cloudcephosd1039 - https://phabricator.wikimedia.org/T394333#10875237 (10Jclark-ctr) a:03Jclark-ctr [12:25:22] 10Quarry: Quarry should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395800 (10taavi) 03NEW [12:25:43] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cloudcephosd10[48-51] & relocate cloudcephosd1039 - https://phabricator.wikimedia.org/T394333#10875250 (10Jclark-ctr) [12:25:57] 10Quarry: Allow SQL queries on result-set tables generated from previous queries - https://phabricator.wikimedia.org/T380206#10875251 (10taavi) p:05Triage→03Low [12:26:31] 06cloud-services-team, 10Cloud-VPS: openstack: keystone may be failing to add users to the bastion project - https://phabricator.wikimedia.org/T379550#10875253 (10Andrew) ` root@cloudcontrol1006:~# openstack role remove --user mortalandrew --project bastion reader root@cloudcontrol1006:~# openstack role assign... [12:28:40] 10Quarry: Remove redis from Quarry - https://phabricator.wikimedia.org/T360584#10875270 (10taavi) 05Open→03Declined Declining given that Redis licensing is OK again. [12:35:47] FIRING: NodeDown: Node cloudcephosd1039 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1039 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [12:53:07] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10875313 (10Ladsgroup) I've dropped the non-term store tables from db1211 and db2243. I think db1211 can become the sanitarium master. [13:10:00] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: Move cloudsw2-d5-eqiad servers to cloudsw1-d5-eqiad - https://phabricator.wikimedia.org/T334644#10875354 (10ayounsi) I see that there are now enough free ports on cloudsw1-d5-eqiad, @Jclark-ctr @dcaro I'm wondering if you could resume the... [13:12:21] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10875377 (10Marostegui) a:03Marostegui [13:19:09] 10Quarry: Quarry should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395800#10875394 (10taavi) [13:19:12] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10875395 (10taavi) [13:20:08] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10875399 (10Marostegui) >>! In T390954#10875313, @Ladsgroup wrote: > I've dropped the non-term store tables from db1211 and db2243. I think db1211 can b... [13:23:57] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10875409 (10Ladsgroup) >>! In T390954#10875399, @Marostegui wrote: >>>! In T390954#10875313, @Ladsgroup wrote: >> I've dropped the non-term store tables... [14:00:58] (03CR) 10Majavah: [C:03+2] Add Bitu container [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318) (owner: 10Slyngshede) [14:03:25] (03Merged) 10jenkins-bot: Add Bitu container [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318) (owner: 10Slyngshede) [14:05:02] 06cloud-services-team, 10Striker, 13Patch-For-Review: Add Bitu container to Striker development environment - https://phabricator.wikimedia.org/T362318#10875587 (10taavi) 05In progress→03Resolved [14:09:27] (03PS7) 10Majavah: Upgrade non-Django dependencies [labs/striker] - 10https://gerrit.wikimedia.org/r/1145823 [14:11:03] (03CR) 10Majavah: [C:03+2] Upgrade non-Django dependencies [labs/striker] - 10https://gerrit.wikimedia.org/r/1145823 (owner: 10Majavah) [14:12:25] (03Merged) 10jenkins-bot: Upgrade non-Django dependencies [labs/striker] - 10https://gerrit.wikimedia.org/r/1145823 (owner: 10Majavah) [14:30:47] FIRING: NodeDown: Node cloudcephosd1039 has been down for long. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1039 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:30:52] 06cloud-services-team: NodeDown Node cloudcephosd1039 has been down for long. - https://phabricator.wikimedia.org/T395811 (10phaultfinder) 03NEW [14:32:13] (03update) 10chuckonwumelu: [build] Return image_name when retrieving build info [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T395035) [14:54:44] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] & relocate cloudcephosd1039 - https://phabricator.wikimedia.org/T394333#10875852 (10Andrew) [14:54:54] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10875853 (10Andrew) [14:55:44] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10875856 (10Andrew) After conversation with @Jclark-ctr we're going to move cloudcephosd1046 (part of T378828 and not yet networked or in service) instead... [15:17:26] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10875936 (10Marostegui) Sanitarium host has been cloned [15:19:59] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10875941 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudcontrol2010-dev.codfw.wmnet with OS boo... [15:25:16] (03open) 10andrew: add 'octavia' service project to hold amphorae [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239 (https://phabricator.wikimedia.org/T393783) [15:29:22] (03update) 10andrew: add 'octavia' service project to hold amphorae [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239 [15:30:45] (03update) 10andrew: add 'octavia' service project to hold amphorae [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239 [15:32:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239 [15:32:20] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239 [15:35:48] (03update) 10andrew: add 'octavia' service project to hold amphorae [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239 [15:36:41] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239 [15:36:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239 [15:37:49] (03merge) 10andrew: add 'octavia' service project to hold amphorae [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/239 [15:38:08] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:41:24] andrew@cloudcumin1001 tofu (PID 1407162) is awaiting input [15:42:42] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Consider setting up an https://github.com/knyar/phalerts instance in metricsinfra - https://phabricator.wikimedia.org/T394446#10875992 (10bd808) >>! In T394446#10874423, @taavi wrote: > @bd808 I'm guessing you're after a rule that sends all d... [15:44:21] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:48:00] (03update) 10andrew: eqiad1: introduce openstack octavia network support [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237 (https://phabricator.wikimedia.org/T394099) (owner: 10aborrero) [15:49:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:49:30] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:24:08] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Privacy Engineering, 07SecTeam-Processed: Add "wikishared" database to wiki replicas - https://phabricator.wikimedia.org/T395072#10876200 (10sbassett) [16:24:36] (03open) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:25:29] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:25:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:25:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:26:17] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:28:25] (03update) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:28:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:28:45] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:29:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:29:15] (03update) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:29:22] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:29:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:30:06] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:31:54] (03update) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:32:44] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:33:03] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [16:35:31] (03open) 10andrew: Draft: Andrewtest [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241 (https://phabricator.wikimedia.org/T393783) [16:35:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241 [16:35:56] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241 [16:36:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241 [16:36:21] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241 [16:36:28] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241 [16:36:47] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241 [16:37:00] (03close) 10andrew: Draft: Andrewtest [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/241 (https://phabricator.wikimedia.org/T393783) [16:40:10] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10876331 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudcontrol2010-dev.codfw.wmnet with OS bookwor... [16:46:40] 10Toolforge (Toolforge iteration 20): [components-api] Add basic prometheus metrics - https://phabricator.wikimedia.org/T394276#10876353 (10Raymond_Ndibe) working now. We have these grafana dashboards. These are probably enough: * https://grafana-rw.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&from... [16:50:07] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10876368 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudcontrol2010-dev.codfw.wmnet with OS boo... [16:55:13] 10Toolforge (Toolforge iteration 20): [components-api] deploy on tools - https://phabricator.wikimedia.org/T394337#10876418 (10Raymond_Ndibe) we can probably mark this as resolved [16:59:24] 06cloud-services-team, 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [jobs-api] Introduce deprecation metrics - https://phabricator.wikimedia.org/T390137#10876434 (10Raymond_Ndibe) [17:15:45] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10876497 (10Jhancock.wm) a:05Jhancock.wm→03Andrew @Andrew not sure why but i can't get it to pxe at all anymore. Can you take a look for me? Thank you! [17:18:56] (03open) 10andrew: Demo change -- do not merge [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242 [17:19:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242 [17:19:30] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242 [17:19:46] (03close) 10andrew: Demo change -- do not merge [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/242 [17:20:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:20:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:21:55] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10876535 (10Jclark-ctr) @Andrew @dcaro Fyi these have Boss cards and are not supported with legacy bios [17:23:08] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10876552 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephosd1048.eqiad.wmnet with OS bullseye [17:23:20] 10Tool-schedule-deployment, 06Security-Team, 07SecTeam-Processed, 07Security, 05Vuln-XSS: https://schedule-deployment.toolforge.org/ should HTML-escape the patch title - https://phabricator.wikimedia.org/T395574#10876562 (10sbassett) [17:24:13] (03update) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:24:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:24:37] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:25:03] 10Cloud-VPS (Quota-requests): Temporary quota increase for 'cvn' - https://phabricator.wikimedia.org/T395274#10876573 (10Krinkle) Ack, I'll start with IPv6 and see how that goes. Currently having trouble getting the code to compile and run correctly on Debian 12, so not currently blocked by this. [17:28:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:28:24] (03update) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:28:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:29:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:29:45] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:32:56] 06cloud-services-team, 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [jobs-api] Introduce deprecation metrics - https://phabricator.wikimedia.org/T390137#10876666 (10Raymond_Ndibe) grafana dashboard can be found here: https://grafana.wmcloud.org/d/kcAb-KUSz/jobs-api?orgId=1&from=2025-05-26T17:30:... [17:33:10] 06cloud-services-team, 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [jobs-api] Introduce deprecation metrics - https://phabricator.wikimedia.org/T390137#10876667 (10Raymond_Ndibe) [17:35:45] (03update) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:35:56] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:36:15] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:36:50] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/246 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:36:51] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/246 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:36:53] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/246 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:37:01] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/68 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:37:02] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/68 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:37:09] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/68 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:37:15] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/67 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:37:19] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/67 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:37:22] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/67 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:38:46] (03update) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:38:49] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:38:52] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/43 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:38:53] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/43 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:38:57] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/43 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:04] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/67 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:04] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/48 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:05] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/48 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:09] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:39:16] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/48 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:18] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/136 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:19] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/136 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:25] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/136 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:26] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/57 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:27] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/57 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:38] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/57 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:40] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.70-20250602173721-d9cff6ee [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 [17:39:41] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/172 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:45] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.70-20250602173721-d9cff6ee [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 [17:39:49] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/172 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:39:54] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/172 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:40:01] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/171 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:40:09] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/171 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:40:27] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/171 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:40:39] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/171 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:40:40] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/23 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:40:45] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/23 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:40:58] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/84 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:41:01] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/23 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:41:03] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/84 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:41:15] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/84 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:41:33] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.71-20250602173924-b6f33a00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 [17:41:36] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.71-20250602173924-b6f33a00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 [17:42:40] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.20-20250602173933-8329610f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/800 [17:43:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:43:47] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/82 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:43:48] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/82 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:43:51] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/82 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:05] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/106 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:06] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:44:08] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/106 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:15] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/104 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:18] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/106 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:19] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/104 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:28] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/35 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:29] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/104 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:33] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/35 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:43] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:44] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/35 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:49] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:58] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/83 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:44:58] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:45:02] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:45:03] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/83 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:45:14] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/83 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:45:20] (03update) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:45:21] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:45:26] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 [17:45:30] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:45:34] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 [17:45:38] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:45:58] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:46:08] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.379-20250602174016-f4c98df1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/802 [17:46:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [17:46:12] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.379-20250602174016-f4c98df1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/802 [17:46:20] (03merge) 10andrew: Update security groups for octavia project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/240 [17:46:24] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:46:39] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.108-20250602174338-58f0be86 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 [17:46:39] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.108-20250602174338-58f0be86 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 [17:46:53] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/171 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:46:55] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [17:47:17] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.193-20250602173948-a871c089 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/804 [17:47:23] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.193-20250602173948-a871c089 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/804 [17:47:36] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/83 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:49:39] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.109-20250602174752-a070a780 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 [17:49:42] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.109-20250602174752-a070a780 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 [17:50:20] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250602173933-8329610f [repos/cloud/toolforge/toolforge-deploy] (bump_api-gateway) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/800 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:50:52] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250602173933-8329610f [repos/cloud/toolforge/toolforge-deploy] (bump_api-gateway) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/800 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:51:08] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] (bump_maintain-harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:51:24] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] (bump_maintain-harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:51:39] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.379-20250602174016-f4c98df1 [repos/cloud/toolforge/toolforge-deploy] (bump_jobs-emailer) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/802 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:51:47] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.379-20250602174016-f4c98df1 [repos/cloud/toolforge/toolforge-deploy] (bump_jobs-emailer) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/802 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:52:01] (03update) 10raymond-ndibe: builds-api: bump to 0.0.193-20250602173948-a871c089 [repos/cloud/toolforge/toolforge-deploy] (bump_jobs-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/804 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:52:10] (03update) 10raymond-ndibe: builds-api: bump to 0.0.193-20250602173948-a871c089 [repos/cloud/toolforge/toolforge-deploy] (bump_jobs-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/804 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:52:28] (03update) 10raymond-ndibe: components-api: bump to 0.0.109-20250602174752-a070a780 [repos/cloud/toolforge/toolforge-deploy] (bump_builds-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:52:39] (03update) 10raymond-ndibe: components-api: bump to 0.0.109-20250602174752-a070a780 [repos/cloud/toolforge/toolforge-deploy] (bump_builds-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:58:35] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.71-20250602173924-b6f33a00 [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:58:42] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.71-20250602173924-b6f33a00 [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:59:52] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [18:01:26] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway [18:03:44] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [18:05:24] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway [18:05:27] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [18:06:30] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [18:06:49] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10876902 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudcephosd1048.eqiad.wmnet with OS bullseye exe... [18:07:16] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10876904 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephosd1048.eqiad.wmnet with OS bullseye [18:20:17] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.71-20250602173924-b6f33a00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:21:47] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [18:35:10] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [18:38:58] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [18:49:30] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [18:58:08] 06cloud-services-team, 10Toolforge, 10Tools: Flickr blocking image requests from Toolforge k8s, breaking multiple tools - https://phabricator.wikimedia.org/T384468#10877047 (10Andrew) ` Hello again! Our tools are now getting a 403 from flicker for every request, which suggests that you are actively blocking... [19:00:22] 10Cloud-VPS (Quota-requests): Pixel project "disk40" flavor, and perhaps a few more cores? - https://phabricator.wikimedia.org/T395837 (10Mhurd) 03NEW [19:08:03] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.71-20250602173924-b6f33a00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:08:08] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.71-20250602173924-b6f33a00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:08:08] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.71-20250602173924-b6f33a00 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/799 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:08:11] (03merge) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [19:08:12] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250602173933-8329610f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/800 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:08:20] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250602173933-8329610f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/800 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:08:24] (03approved) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250602173933-8329610f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/800 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:08:28] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250602173933-8329610f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/800 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:08:46] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] (bump_maintain-harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:08:48] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] (bump_maintain-harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:08:54] (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] (bump_maintain-harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:09:06] (03merge) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250602173933-8329610f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/800 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:09:09] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:11:53] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:12:15] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:12:30] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:12:39] (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.58-20250602174121-4511de9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/801 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:12:41] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.379-20250602174016-f4c98df1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/802 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:12:54] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.379-20250602174016-f4c98df1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/802 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:12:55] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.379-20250602174016-f4c98df1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/802 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:13:00] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.379-20250602174016-f4c98df1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/802 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:15:23] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.379-20250602174016-f4c98df1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/802 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:15:25] (03update) 10raymond-ndibe: builds-api: bump to 0.0.193-20250602173948-a871c089 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/804 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:15:37] (03update) 10raymond-ndibe: builds-api: bump to 0.0.193-20250602173948-a871c089 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/804 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:15:39] (03approved) 10raymond-ndibe: builds-api: bump to 0.0.193-20250602173948-a871c089 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/804 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:15:41] (03update) 10raymond-ndibe: builds-api: bump to 0.0.193-20250602173948-a871c089 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/804 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:15:57] (03merge) 10raymond-ndibe: builds-api: bump to 0.0.193-20250602173948-a871c089 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/804 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:16:00] (03update) 10raymond-ndibe: components-api: bump to 0.0.109-20250602174752-a070a780 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:18:47] (03update) 10raymond-ndibe: components-api: bump to 0.0.109-20250602174752-a070a780 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:18:49] (03approved) 10raymond-ndibe: components-api: bump to 0.0.109-20250602174752-a070a780 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:18:52] (03update) 10raymond-ndibe: components-api: bump to 0.0.109-20250602174752-a070a780 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:19:08] (03merge) 10raymond-ndibe: components-api: bump to 0.0.109-20250602174752-a070a780 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/803 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:49:51] (03PS1) 10BryanDavis: Update security status check for upstream changes [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1152811 (https://phabricator.wikimedia.org/T301082) [19:53:30] (03PS2) 10BryanDavis: Update security status check for upstream changes [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1152811 (https://phabricator.wikimedia.org/T301082) [19:55:20] (03CR) 10BryanDavis: [V:03+1 C:03+2] Update security status check for upstream changes [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1152811 (https://phabricator.wikimedia.org/T301082) (owner: 10BryanDavis) [19:55:51] (03Merged) 10jenkins-bot: Update security status check for upstream changes [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1152811 (https://phabricator.wikimedia.org/T301082) (owner: 10BryanDavis) [20:17:49] (03PS1) 10BryanDavis: Upgrade to mwclient 0.11.0 [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1152815 (https://phabricator.wikimedia.org/T372309) [20:18:58] (03CR) 10BryanDavis: [C:03+2] Upgrade to mwclient 0.11.0 [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1152815 (https://phabricator.wikimedia.org/T372309) (owner: 10BryanDavis) [20:19:28] (03Merged) 10jenkins-bot: Upgrade to mwclient 0.11.0 [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1152815 (https://phabricator.wikimedia.org/T372309) (owner: 10BryanDavis) [20:35:28] 10Striker, 10Tool-phab-ban, 10Bitu, 10MediaWiki-Action-API: Removal of writeapi from siteinfo output breaks all mwclient-based bots, including stashbot (Server Admin Log) - https://phabricator.wikimedia.org/T371977#10877438 (10bd808) [20:42:21] 10Tool-phab-ban: Upgrade to mwclient 0.11.0 - https://phabricator.wikimedia.org/T372310#10877450 (10bd808) 05Open→03Resolved a:03bd808 The dependency is floating in requirements.txt, so we just need to update the active venv. `lang=shell-session $ ssh dev.toolforge.org tools-bastion-12.tools.eqiad1:~$... [20:42:44] 10Striker, 10Bitu, 10MediaWiki-Action-API: Removal of writeapi from siteinfo output breaks all mwclient-based bots, including stashbot (Server Admin Log) - https://phabricator.wikimedia.org/T371977#10877455 (10bd808) [20:43:56] 06cloud-services-team, 10Striker: Upgrade to mwclient 0.11.0 - https://phabricator.wikimedia.org/T372313#10877456 (10taavi) a:03taavi [20:46:37] 06cloud-services-team, 10Striker: Upgrade to mwclient 0.11.0 - https://phabricator.wikimedia.org/T372313#10877458 (10bd808) 05Open→03Resolved https://gerrit.wikimedia.org/r/c/labs/striker/+/1145823 [21:17:00] (03update) 10andrew: eqiad1: introduce openstack octavia network support [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237 (https://phabricator.wikimedia.org/T394099) (owner: 10aborrero) [21:23:54] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877622 (10Andrew) @Jclark-ctr, the new preseed recipe seems to work ok, 1048 is now reimaging properly. 1049 failed for me in a totally different way but... [21:25:33] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877624 (10Jclark-ctr) @Andrew thanks i was looking at 1048 right now also i see it imaging! yea i have not adjusted a few settings for the rest work on... [21:33:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237 [21:33:23] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237 [21:34:34] (03merge) 10andrew: eqiad1: introduce openstack octavia network support [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/237 (https://phabricator.wikimedia.org/T394099) (owner: 10aborrero) [21:34:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [21:35:42] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [21:46:31] 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-api] "No such image" error when running with prebuilt image - https://phabricator.wikimedia.org/T395533#10877680 (10Raymond_Ndibe) update: seems like we tried to disable `prebuilt` as part of the MVP release (but that didn't work since we we... [21:47:40] (03update) 10raymond-ndibe: [components.models.api_models] add config version check [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/74 (https://phabricator.wikimedia.org/T394273) [21:47:46] (03approved) 10raymond-ndibe: [components.models.api_models] add config version check [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/74 (https://phabricator.wikimedia.org/T394273) [21:48:30] (03update) 10raymond-ndibe: [toolforge_models] update toolforge_models.py [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/79 [21:49:02] (03approved) 10raymond-ndibe: [toolforge_models] update toolforge_models.py [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/79 [21:52:14] (03merge) 10raymond-ndibe: [toolforge_models] update toolforge_models.py [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/79 [21:52:19] (03update) 10raymond-ndibe: [deploy] support health-checks and port [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/75 (https://phabricator.wikimedia.org/T362072) [21:52:36] (03update) 10raymond-ndibe: [components.models.api_models] add config version check [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/74 (https://phabricator.wikimedia.org/T394273) [21:53:45] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877694 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephosd1049.eqiad.wmnet with OS bullseye [21:54:27] (03merge) 10raymond-ndibe: [components.models.api_models] add config version check [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/74 (https://phabricator.wikimedia.org/T394273) [21:54:53] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.110-20250602215227-7502d56b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 [21:55:44] (03update) 10raymond-ndibe: [components.tool_router] add example config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/82 (https://phabricator.wikimedia.org/T394753) [21:57:14] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.111-20250602215438-da42aed6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 (https://phabricator.wikimedia.org/T394273) [21:57:16] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.111-20250602215438-da42aed6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 (https://phabricator.wikimedia.org/T394273) [22:01:33] (03PS1) 10Andrew Bogott: Move octavia secrets into deployment-specific subdirs, add for eqiad [labs/private] - 10https://gerrit.wikimedia.org/r/1152839 (https://phabricator.wikimedia.org/T393783) [22:01:34] (03PS1) 10Andrew Bogott: Correct the name of a fake octavia password [labs/private] - 10https://gerrit.wikimedia.org/r/1152840 (https://phabricator.wikimedia.org/T393783) [22:01:58] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877711 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephosd1050.eqiad.wmnet with OS bullseye [22:06:28] (03CR) 10Andrew Bogott: [C:03+2] Move octavia secrets into deployment-specific subdirs, add for eqiad [labs/private] - 10https://gerrit.wikimedia.org/r/1152839 (https://phabricator.wikimedia.org/T393783) (owner: 10Andrew Bogott) [22:06:33] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Correct the name of a fake octavia password [labs/private] - 10https://gerrit.wikimedia.org/r/1152840 (https://phabricator.wikimedia.org/T393783) (owner: 10Andrew Bogott) [22:06:37] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Move octavia secrets into deployment-specific subdirs, add for eqiad [labs/private] - 10https://gerrit.wikimedia.org/r/1152839 (https://phabricator.wikimedia.org/T393783) (owner: 10Andrew Bogott) [22:29:11] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877774 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephosd1051.eqiad.wmnet with OS bullseye [22:45:05] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877804 (10Jclark-ctr) [23:10:29] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877836 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudcephosd1049.eqiad.wmnet with OS bullseye com... [23:14:16] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [23:19:25] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [23:23:03] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877843 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudcephosd1050.eqiad.wmnet with OS bullseye com... [23:24:55] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877844 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudcephosd1051.eqiad.wmnet with OS bullseye com... [23:25:27] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10877846 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephosd1048.eqiad.wmnet with OS bullseye