[00:12:45] (03open) 10bd808: bin/cron: Organize output by year and month [toolforge-repos/jenkins-build-stats] - 10https://gitlab.wikimedia.org/toolforge-repos/jenkins-build-stats/-/merge_requests/2 [00:13:05] (03merge) 10bd808: bin/cron: Organize output by year and month [toolforge-repos/jenkins-build-stats] - 10https://gitlab.wikimedia.org/toolforge-repos/jenkins-build-stats/-/merge_requests/2 [00:16:09] FIRING: CephSlowOps: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [00:16:14] 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 1 slow ops - https://phabricator.wikimedia.org/T380239 (10phaultfinder) 03NEW [00:18:07] 06cloud-services-team, 10Cloud-VPS, 10WikiWho: Enable use of web proxy for wikiwho.net domain - https://phabricator.wikimedia.org/T376637#10334234 (10MusikAnimal) [00:19:02] 06cloud-services-team, 10Cloud-VPS, 10WikiWho: Enable use of web proxy for wikiwho.net domain - https://phabricator.wikimedia.org/T376637#10334231 (10MusikAnimal) I have received confirmation that they're interested in the vanity domain solution: ` … their admins should please set up a “vanity domain” for u... [00:21:09] RESOLVED: CephSlowOps: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [00:22:20] 06cloud-services-team, 10Cloud-VPS, 10WikiWho: Enable use of web proxy for wikiwho.net domain - https://phabricator.wikimedia.org/T376637#10334254 (10MusikAnimal) [02:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:15:28] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate True, for hosts list: ['cloudvirt1062'] [04:15:56] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate True, for hosts list: ['cloudvirt1062'] [04:27:06] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate True, for hosts list: ['cloudvirt1062'] [04:27:28] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate True, for hosts list: ['cloudvirt1062'] [04:39:03] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#10334598 (10Andrew) [04:40:05] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#10334595 (10Andrew) 05In progress→03Resolved Finally moved the last one of these, cloudvirt1062 [05:54:14] FIRING: Kernel error: Server cloudvirt1062 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudvirt1062 - https://alerts.wikimedia.org/?q=alertname%3DKernel+error [05:54:14] FIRING: Kernel warning: Server cloudvirt1062 may have kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Kernel_panic - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-panic-detector?orgId=1&var-instance=cloudvirt1062 - https://alerts.wikimedia.org/?q=alertname%3DKernel+warning [05:54:22] 06cloud-services-team: Kernel error Server cloudvirt1062 may have kernel errors - https://phabricator.wikimedia.org/T380249 (10phaultfinder) 03NEW [08:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:04:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:07:30] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: ingress-admission: bump to 0.0.54-20241119090358-300b9ae5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/602 (https://phabricator.wikimedia.org/T362867) [09:07:35] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: ingress-admission: bump to 0.0.54-20241119090358-300b9ae5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/602 (https://phabricator.wikimedia.org/T362867) [09:10:25] 06cloud-services-team: Kernel error Server cloudvirt1062 may have kernel errors - https://phabricator.wikimedia.org/T380249#10334919 (10aborrero) 05Open→03Resolved a:03aborrero server was rebooted: `lang=shell-session aborrero@cloudvirt1062:~$ sudo journalctl -k --prio err Nov 19 04:11:32 cloudvirt106... [09:12:15] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) [09:13:20] (03open) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [09:14:46] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [09:18:01] (03open) 10dcaro: functional_tests: add extra_args to the webservice template [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/603 (https://phabricator.wikimedia.org/T380141) [09:21:41] (03update) 10dcaro: functional_tests: add extra_args to the webservice template [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/603 (https://phabricator.wikimedia.org/T380141) [09:22:51] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) [09:30:44] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) [09:32:47] 06cloud-services-team, 06Infrastructure-Foundations, 10SRE-tools, 07IPv6: Some WMCS clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271139#10334985 (10Volans) I think it can be resolved, list updated as of today: ` an-redacteddb1001.eqiad.wmnet,clouddb2002-dev.codfw.wmnet,cloud... [09:33:41] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: nova-fullstack failing in codfw1dev - https://phabricator.wikimedia.org/T380208#10334964 (10aborrero) 05Open→03In progress p:05Triage→03Medium [09:34:19] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Some WMCS clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts - https://phabricator.wikimedia.org/T312557#10334981 (10Volans) 05Stalled→03Resolved a:03Volans There are no more inconsistent cluster, all the above have been resolve... [09:35:28] (03update) 10dcaro: functional_tests: add extra_args to the webservice template [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/603 (https://phabricator.wikimedia.org/T380141) [09:40:19] (03update) 10dcaro: functional_tests: add extra_args to the webservice template [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/603 (https://phabricator.wikimedia.org/T380141) [09:42:13] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [09:49:14] (03open) 10dcaro: jobs-emailer: use shorter loop times for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/604 [09:54:47] (03approved) 10sstefanova: functional_tests: add extra_args to the webservice template [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/603 (https://phabricator.wikimedia.org/T380141) (owner: 10dcaro) [09:58:00] (03merge) 10dcaro: functional_tests: add extra_args to the webservice template [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/603 (https://phabricator.wikimedia.org/T380141) [10:09:36] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice [10:09:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:09:50] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [10:10:03] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice [10:10:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:11:09] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [10:11:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:11:15] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [10:11:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:13:21] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [10:13:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:13:26] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [10:13:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:13:51] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [10:13:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:23:59] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [10:24:02] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [10:24:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:24:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:24:07] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [10:24:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:24:24] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [10:24:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:24:28] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [10:24:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:24:37] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [10:25:47] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [10:25:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:25:54] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [10:25:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:26:22] !log aborrero@cloudcumin2001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [10:27:08] (03PS1) 10David Caro: toolforge.component.deploy: fix when there's many forks with same name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1092784 [10:27:35] (03PS2) 10David Caro: create_project: add a bit nicer example [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1090889 [10:28:58] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: nova-fullstack failing in codfw1dev - https://phabricator.wikimedia.org/T380208#10335154 (10aborrero) it seems the problem is nova is not finding hypervisors: `'No valid host was found. There are not enough hosts available.',` [10:29:42] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: nova-fullstack failing in codfw1dev - https://phabricator.wikimedia.org/T380208#10335156 (10aborrero) `lang=shell-session aborrero@cloudcontrol2005-dev:~ $ sudo wmcs-openstack hypervisor list +--------------------------------------+----------------------... [10:31:05] 06cloud-services-team, 10Cloud-VPS, 07Puppet: Preserve formatting and comments etc. in ENC Hiera - https://phabricator.wikimedia.org/T250622#10335164 (10dcaro) Currently not supported by the pyyaml https://github.com/yaml/pyyaml/issues/90 [10:32:01] (03update) 10dcaro: jobs-emailer: pull always on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/600 [10:32:39] (03PS1) 10Muehlenhoff: airflow_platform_eng: Add missing stub secret [labs/private] - 10https://gerrit.wikimedia.org/r/1092785 (https://phabricator.wikimedia.org/T378443) [10:33:39] (03CR) 10David Caro: create_project: add a bit nicer example [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1090889 (owner: 10David Caro) [10:33:42] (03CR) 10David Caro: [C:03+2] create_project: add a bit nicer example [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1090889 (owner: 10David Caro) [10:37:26] (03Merged) 10jenkins-bot: create_project: add a bit nicer example [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1090889 (owner: 10David Caro) [10:38:00] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [jobs-emailer] http requests are blocked by the loops - https://phabricator.wikimedia.org/T379924#10335185 (10dcaro) 05Open→03In progress [10:42:23] (03CR) 10Slyngshede: [C:03+1] "Lgtm" [labs/private] - 10https://gerrit.wikimedia.org/r/1092785 (https://phabricator.wikimedia.org/T378443) (owner: 10Muehlenhoff) [10:57:37] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: nova-fullstack failing in codfw1dev - https://phabricator.wikimedia.org/T380208#10335220 (10aborrero) I see this rabbitmq crash with `aborrero@cloudcontrol2005-dev:~$ sudo tail -f /var/log/rabbitmq/rabbit@rabbitmq03.codfw1dev.wikimediacloud.org.log` `li... [10:58:10] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: rabbitmq is crashing - https://phabricator.wikimedia.org/T380208#10335223 (10aborrero) [11:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:12:00] (03CR) 10FNegri: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1092784 (owner: 10David Caro) [11:14:59] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: rabbitmq is crashing - https://phabricator.wikimedia.org/T380208#10335247 (10aborrero) `lang=shell-session aborrero@cloudcontrol2005-dev:~ $ sudo rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().' 2024-11-19 11:13:37 There are 5900 processes... [11:15:30] (03CR) 10David Caro: [C:03+2] toolforge.component.deploy: fix when there's many forks with same name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1092784 (owner: 10David Caro) [11:16:12] (03open) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [11:16:15] (03open) 10dcaro: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) [11:17:39] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: rabbitmq is crashing - https://phabricator.wikimedia.org/T380208#10335252 (10aborrero) rabbitmq-server on cloudcontrol2006-dev had severe problems: `lang=shell-session aborrero@cloudcontrol2006-dev:~ $ sudo rabbitmqctl eval 'rabbit_diagnostic... [11:18:51] (03Merged) 10jenkins-bot: toolforge.component.deploy: fix when there's many forks with same name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1092784 (owner: 10David Caro) [11:20:49] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] airflow_platform_eng: Add missing stub secret [labs/private] - 10https://gerrit.wikimedia.org/r/1092785 (https://phabricator.wikimedia.org/T378443) (owner: 10Muehlenhoff) [11:33:18] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: rabbitmq is crashing - https://phabricator.wikimedia.org/T380208#10335283 (10aborrero) the cluster is just in split brain mode: `lang=shell-session aborrero@cloudcumin2001:~ $ sudo cumin cloudcontrol200[4-6]* 'rabbitmqctl cluster_status | gre... [12:39:31] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [12:49:00] (03open) 10raymond-ndibe: [toolforge-deploy] allow for running both admin and tools tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/605 (https://phabricator.wikimedia.org/T358225) [12:52:01] (03update) 10raymond-ndibe: [toolforge-deploy] allow for running both admin and tools tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/605 (https://phabricator.wikimedia.org/T358225) [12:54:32] (03update) 10raymond-ndibe: [toolforge-deploy] allow for running both admin and tools tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/605 (https://phabricator.wikimedia.org/T358225) [12:55:18] (03update) 10raymond-ndibe: [toolforge-deploy] allow for running both admin and tools tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/605 (https://phabricator.wikimedia.org/T358225) [13:00:49] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [13:06:17] (03update) 10dcaro: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) [13:09:00] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [13:13:49] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [13:14:33] (03update) 10dcaro: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) [13:18:26] (03open) 10dcaro: bump_version: copy from webservice [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/95 [13:20:47] (03open) 10dcaro: bump_version: copy from webservice [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/68 [13:28:14] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [13:34:02] (03approved) 10sstefanova: bump_version: copy from webservice [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/68 (owner: 10dcaro) [13:34:35] (03open) 10dcaro: jobs-api: fix the debug setting [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/606 [13:35:00] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [13:36:12] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: rabbitmq is crashing - https://phabricator.wikimedia.org/T380208#10335640 (10aborrero) I needed to do a full reset of the cluster, mostly following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/RabbitMQ#Resetting_the_HA_setup [13:37:03] !log aborrero@cloudcumin2001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [13:37:33] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [13:37:58] (03merge) 10dcaro: bump_version: copy from webservice [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/68 [13:38:59] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [13:42:22] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [13:43:25] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [13:43:50] !log aborrero@cloudcumin2001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [13:48:07] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [13:52:56] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: rabbitmq is crashing - https://phabricator.wikimedia.org/T380208#10335704 (10aborrero) Despite the full reset, having corrected the split brain, I still see nova-compute complaining. `AMQP server on rabbitmq01.codfw1dev.wikimediacloud.org:567... [13:56:45] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [14:04:51] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [14:08:48] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [14:14:33] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10335805 (10fnegri) Current situation: * tools-db-1: primary, running MariaDB 10.4 * tools-db-3: replica, running MariaDb 10.4 * tools-db-4: n... [14:20:50] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [14:23:32] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [14:30:08] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [14:34:43] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [14:34:56] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [14:38:10] (03update) 10dcaro: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) [14:38:52] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [14:39:25] (03update) 10dcaro: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) [14:39:55] (03update) 10dcaro: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) [14:53:48] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [14:54:49] (03approved) 10sstefanova: bump_version: copy from webservice [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/95 (owner: 10dcaro) [14:56:12] (03merge) 10dcaro: bump_version: copy from webservice [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/95 [14:58:56] (03open) 10dcaro: hartman/rangeerror [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/116 [14:59:30] (03update) 10dcaro: Fix error in [A-z] range [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/116 [15:02:58] (03open) 10dcaro: bump_version: copy from webservice [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/77 [15:04:18] (03approved) 10sstefanova: bump_version: copy from webservice [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/77 (owner: 10dcaro) [15:06:52] (03merge) 10dcaro: bump_version: copy from webservice [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/77 [15:09:23] (03open) 10dcaro: utils: copy scripts from webservice [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/34 [15:13:13] (03open) 10dcaro: toolforge_deploy_mr: add clis to the bash completion [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/215 [15:15:57] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [15:21:15] (03approved) 10dcaro: Fix error in [A-z] range [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/116 [15:21:15] (03merge) 10dcaro: Fix error in [A-z] range [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/116 [15:21:15] (03merge) 10dcaro: Fix error in [A-z] range [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/115 (owner: 10hartman) [15:23:01] 10Toolforge (Toolforge iteration 16): [components-api] Limit the amount of deployments to (say) 25 - https://phabricator.wikimedia.org/T380283 (10dcaro) 03NEW [15:23:18] 10Toolforge (Toolforge iteration 16): [components-api] Limit the amount of deployments to (say) 25 - https://phabricator.wikimedia.org/T380283#10336202 (10dcaro) [15:23:21] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10336203 (10dcaro) [15:23:33] 10Toolforge (Toolforge iteration 16): [components-api] Limit the amount of deployments to (say) 25 - https://phabricator.wikimedia.org/T380283#10336205 (10dcaro) p:05Triage→03Medium [15:24:02] 10Toolforge (Toolforge iteration 16): [components-api] Limit the amount of deployments to (say) 25 - https://phabricator.wikimedia.org/T380283#10336206 (10dcaro) [15:24:27] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [15:25:54] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) [15:27:13] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-api: bump to 0.0.176-20241119152031-5523b049 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/607 [15:27:16] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-api: bump to 0.0.176-20241119152031-5523b049 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/607 [15:31:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:31:53] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [15:37:38] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [15:46:16] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:49:17] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [15:52:04] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [15:56:21] 06cloud-services-team: Kernel error Server cloudvirt1062 may have kernel errors - https://phabricator.wikimedia.org/T380249#10336451 (10Andrew) I reimaged it last night to move off of linuxbridge. [16:07:12] 10tool-wscontest, 07good first task: Add UTC in the WSContest contest page - https://phabricator.wikimedia.org/T331225#10336560 (10AS1100K) I am new to Phabricator and has never used it before and confused why my [[ https://github.com/wikisource/wscontest/pull/76 | PR #76 ]] on GitHub isn't referenced here. Si... [16:11:03] (03approved) 10dcaro: builds-api: bump to 0.0.176-20241119152031-5523b049 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/607 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:11:04] (03update) 10dcaro: builds-api: bump to 0.0.176-20241119152031-5523b049 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/607 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:11:06] (03merge) 10dcaro: builds-api: bump to 0.0.176-20241119152031-5523b049 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/607 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:28:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [16:31:08] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [16:49:54] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10336758 (10Rodrigo) |**Wikitech account/LDAP:**| Rodrigo| |**SUL account**| Rodrigo| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |Y| |**I have visited [[ https://wikitech.w... [17:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:21:28] 10Tools: Update welcome message in Zulip's goodbot - https://phabricator.wikimedia.org/T310826#10336920 (10Aklapper) a:05QEDK→03None @QEDK: Removing task assignee as this open task has been assigned for more than two years - See the email sent to task assignee on October 11th. Please assign this task to your... [17:27:03] 10Tool-infogeek244-grammabot: Incorrect code repository information for infogeek244-grammabot - https://phabricator.wikimedia.org/T315151#10336968 (10Aklapper) a:05Infogeek244→03None @Infogeek244: Removing task assignee as this open task has been assigned for more than two years - See the email sent to task... [17:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:31:29] 14Grid-Engine-to-K8s-Migration, 10Tools: Migrate multichill from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319912#10336998 (10Aklapper) a:05Multichill→03None @Multichill: Removing task assignee as this open task has been assigned for more than two years - See the ema... [17:38:43] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10337108 (10Andrew) [17:40:37] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev - https://phabricator.wikimedia.org/T380303 (10Andrew) 03NEW [17:52:15] 06cloud-services-team, 10Toolforge: add on-wiki edits of toolforge tools to toolviews report - https://phabricator.wikimedia.org/T317953#10337291 (10Aklapper) a:05Raymond_Ndibe→03None @Raymond_Ndibe: Removing task assignee as this open task has been assigned for more than two years - See the email sent to... [17:55:50] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev - https://phabricator.wikimedia.org/T380303#10337334 (10Andrew) This is due to https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/commit/ca8b21897b2e084197b9ea85dbaf17f109661fa8 [17:58:56] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade to v16 - https://phabricator.wikimedia.org/T306820#10337399 (10Aklapper) a:05dcaro→03None @dcaro: Removing task assignee as this open task has been assig... [18:00:43] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [alerting] Use the `services` label in cookbook to silence groups of alerts - https://phabricator.wikimedia.org/T318653#10337396 (10Aklapper) a:05dcaro→03None @dcaro: Removing task assignee as this open... [18:01:34] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [alerting][cookbooks] When removing silences, give some time for the alerts to clear - https://phabricator.wikimedia.org/T318654#10337408 (10Aklapper) a:05dcaro→03None @dcaro: Removing task assignee as... [18:02:08] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: cookbooks.ceph: replace/modify --force to allow specifying the osd flags you don't want to take into account when checking the cluster health - https://phabricator.wikimedia.org/T315422#10337403 (10Aklapper... [18:03:51] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev - https://phabricator.wikimedia.org/T380303#10337440 (10Andrew) Here are the orphaned VMs: [] f93ed978bc5943c4b9c3fb1bfc70d3b4: f950d7b4-ac86-4d12-b19a-096c0d52cd19 [] f93ed978bc5943c4b9c3fb1bfc70d3... [18:08:50] (03CR) 10Ladsgroup: [C:03+2] Index research/recommendation-api under services [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1092326 (https://phabricator.wikimedia.org/T379761) (owner: 10Sbisson) [18:09:49] (03CR) 10Ladsgroup: [C:03+2] Index releng/dev-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1091896 (https://phabricator.wikimedia.org/T380133) (owner: 10Daimona Eaytoy) [18:09:55] (03Merged) 10jenkins-bot: Index research/recommendation-api under services [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1092326 (https://phabricator.wikimedia.org/T379761) (owner: 10Sbisson) [18:10:42] (03Merged) 10jenkins-bot: Index releng/dev-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1091896 (https://phabricator.wikimedia.org/T380133) (owner: 10Daimona Eaytoy) [18:18:28] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310 (10Andrew) 03NEW [18:21:58] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev - https://phabricator.wikimedia.org/T380303#10337684 (10Andrew) Hmph, 84864e4b-5ac8-4c56-bbc8-3a0dbeaea1cb is wedged in a way that was harmless until andrewtestproject was deleted and now I need to d... [18:34:01] (03update) 10dcaro: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) [19:07:42] (03update) 10raymond-ndibe: [registry-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/17 (https://phabricator.wikimedia.org/T362867) [19:07:48] (03update) 10raymond-ndibe: [ingress-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T362867) [19:07:56] (03update) 10raymond-ndibe: [envvars-api] k8s.io deps to 0.28.14 [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/48 (https://phabricator.wikimedia.org/T362867) [19:09:25] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: registry-admission: bump to 0.0.54-20241119190632-ec34b7a8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/608 (https://phabricator.wikimedia.org/T362867) [19:10:50] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: envvars-api: bump to 0.0.62-20241119190711-285eda5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/609 (https://phabricator.wikimedia.org/T362867) [19:16:23] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission (T362867) [19:16:26] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [19:17:32] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission (T362867) [19:17:45] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission (T362867) [19:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:23:19] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission (T362867) [19:23:23] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [19:23:58] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission (T362867) [19:24:13] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362867) [19:25:25] (03update) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [19:25:30] (03approved) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [19:30:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362867) [19:30:18] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [19:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:30:46] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission (T362867) [19:31:59] (03update) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [19:32:16] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362867) [19:32:52] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362867) [19:33:52] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.54-20241119090358-300b9ae5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/602 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:33:55] (03approved) 10raymond-ndibe: ingress-admission: bump to 0.0.54-20241119090358-300b9ae5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/602 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:37:45] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362867) [19:37:50] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [19:38:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362867) [19:41:41] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362867) [19:47:22] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362867) [19:47:26] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [19:53:40] (03approved) 10raymond-ndibe: registry-admission: bump to 0.0.54-20241119190632-ec34b7a8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/608 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:53:51] (03merge) 10raymond-ndibe: registry-admission: bump to 0.0.54-20241119190632-ec34b7a8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/608 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:54:31] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T362867) [19:54:36] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [20:00:17] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T362867) [20:00:22] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [20:01:40] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T362867) [20:02:04] (03update) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [20:02:24] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.54-20241119090358-300b9ae5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/602 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [20:02:43] (03merge) 10raymond-ndibe: ingress-admission: bump to 0.0.54-20241119090358-300b9ae5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/602 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [20:04:05] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico (T362867) [20:05:11] (03update) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [20:07:52] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T362867) [20:07:57] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [20:09:45] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T362867) [20:12:15] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico (T362867) [20:12:35] (03merge) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [20:12:58] (03update) 10raymond-ndibe: jobs-emailer: use shorter loop times for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/604 (owner: 10dcaro) [20:13:18] (03approved) 10raymond-ndibe: jobs-emailer: use shorter loop times for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/604 (owner: 10dcaro) [20:13:26] (03approved) 10raymond-ndibe: jobs-emailer: pull always on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/600 (owner: 10dcaro) [20:13:30] (03update) 10raymond-ndibe: jobs-emailer: pull always on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/600 (owner: 10dcaro) [20:14:23] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api (T362867) [20:14:27] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [20:17:07] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T362867) [20:20:24] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api (T362867) [20:20:28] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [20:28:02] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api (T362867) [20:28:08] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [20:28:22] (03approved) 10raymond-ndibe: [toolforge-deploy] update calico kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585 (https://phabricator.wikimedia.org/T362867) [20:28:26] (03update) 10raymond-ndibe: [toolforge-deploy] update calico kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585 (https://phabricator.wikimedia.org/T362867) [20:28:47] (03merge) 10raymond-ndibe: [toolforge-deploy] update calico kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585 (https://phabricator.wikimedia.org/T362867) [20:29:07] (03approved) 10raymond-ndibe: jobs-api: fix the debug setting [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/606 (owner: 10dcaro) [20:29:55] (03update) 10raymond-ndibe: jobs-emailer: pull always on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/600 (owner: 10dcaro) [20:30:03] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api (T362867) [20:30:42] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [20:31:14] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api (T362867) [20:31:17] !log raymond-ndibe@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api (T362867) [20:31:26] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [20:36:48] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer [20:38:08] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [20:38:36] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10338210 (10aborrero) I think we can consider the example you mention a human mistake. we can easily prevent tofu from deleting resources: https://opentofu.org/docs/v1.7/language/meta-... [20:38:48] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api [20:43:44] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [20:48:36] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [20:53:10] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [20:53:27] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer [20:54:13] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics [20:59:27] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [21:05:32] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [21:05:54] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [21:05:55] (03update) 10sstefanova: cli: add config subcommands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T379091) [21:06:26] (03merge) 10raymond-ndibe: jobs-emailer: pull always on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/600 (owner: 10dcaro) [21:11:45] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [21:18:45] (03approved) 10raymond-ndibe: envvars-api: bump to 0.0.62-20241119190711-285eda5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/609 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [21:18:48] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.62-20241119190711-285eda5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/609 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [21:19:05] (03merge) 10raymond-ndibe: envvars-api: bump to 0.0.62-20241119190711-285eda5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/609 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [21:19:14] (03update) 10raymond-ndibe: jobs-emailer: use shorter loop times for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/604 (owner: 10dcaro) [21:19:34] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [21:25:17] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [21:30:54] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [21:36:38] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [21:37:53] (03merge) 10raymond-ndibe: jobs-emailer: use shorter loop times for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/604 (owner: 10dcaro) [21:38:18] (03update) 10raymond-ndibe: jobs-api: fix the debug setting [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/606 (owner: 10dcaro) [21:38:53] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [21:44:20] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [21:46:25] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [21:52:23] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [21:56:17] (03merge) 10raymond-ndibe: jobs-api: fix the debug setting [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/606 (owner: 10dcaro) [23:16:47] (03open) 10raymond-ndibe: [volume-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/21 (https://phabricator.wikimedia.org/T362867) [23:57:46] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225)