[00:51:47] 10Tool-openstack-browser: openstack-browser support for heat/magnum - https://phabricator.wikimedia.org/T325465#10716178 (10Andrew) 05Open→03Resolved a:03Andrew We now display the upstream magnum UI. It's not great, but that's documented in T343362 [02:12:14] (03merge) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250403032838-697134e5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/729 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [02:16:55] (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (refactor_imagename_field_in_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [02:17:40] (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] (move_most_custom_validations_out_of_api_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T359804) [02:33:08] (03close) 10raymond-ndibe: d/changelog: bump to 16.1.9 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/92 (https://phabricator.wikimedia.org/T359804) [02:33:14] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.9 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/92 (https://phabricator.wikimedia.org/T359804) [02:34:36] (03update) 10raymond-ndibe: [jobs-cli] use imagename in definedjob [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T389118) [02:37:01] (03update) 10raymond-ndibe: [jobs-cli] remove unused variable [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/91 [02:37:06] (03update) 10raymond-ndibe: [jobs-cli] remove unused variable [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/91 [02:44:52] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.9 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T359804) [02:58:51] (03open) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 [02:59:33] (03update) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 [03:05:46] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.9 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T359804) [03:30:26] (03update) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 [03:31:30] (03update) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 [03:31:45] (03update) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 [03:40:08] (03update) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 [03:42:30] (03update) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 [03:46:51] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.9 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T359804) [03:49:18] (03update) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 [03:54:20] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [gitlab-ci] use twine 6.1.0 - https://phabricator.wikimedia.org/T391225 (10Raymond_Ndibe) 03NEW [03:55:03] (03update) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 (https://phabricator.wikimedia.org/T391225) [03:55:48] (03update) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 (https://phabricator.wikimedia.org/T391225) [03:57:11] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [gitlab-ci] use twine 6.1.0 - https://phabricator.wikimedia.org/T391225#10716268 (10Raymond_Ndibe) [03:57:13] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [gitlab-ci] use twine 6.1.0 - https://phabricator.wikimedia.org/T391225#10716270 (10Raymond_Ndibe) 05Open→03In progress [03:57:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [gitlab-ci] use twine 6.1.0 - https://phabricator.wikimedia.org/T391225#10716273 (10Raymond_Ndibe) a:03Raymond_Ndibe [03:58:00] (03approved) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 (https://phabricator.wikimedia.org/T391225) [03:58:04] (03merge) 10raymond-ndibe: [gitlab-ci] twine use latest version [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/49 (https://phabricator.wikimedia.org/T391225) [03:58:29] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [gitlab-ci] use twine 6.1.0 - https://phabricator.wikimedia.org/T391225#10716276 (10Raymond_Ndibe) 05In progress→03Resolved [03:59:29] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.9 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T359804) [04:04:47] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [04:15:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [05:31:06] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [05:42:55] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [06:00:40] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.9 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T359804) [06:00:41] (03approved) 10raymond-ndibe: d/changelog: bump to 16.1.9 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T359804) [06:00:47] (03merge) 10raymond-ndibe: d/changelog: bump to 16.1.9 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T359804) [06:16:26] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] Split the API, core, and storage and runtime models - https://phabricator.wikimedia.org/T359808#10716401 (10Raymond_Ndibe) >>! In T359808#10656079, @taavi wrote: > This doesn't seem to have had any progress since last May - why is it bouncing... [06:41:28] (03update) 10raymond-ndibe: [jobs-cli] use imagename in definedjob [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T389118) [06:45:12] (03update) 10raymond-ndibe: [jobs-cli] use imagename in definedjob [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T389118) [06:51:18] (03approved) 10raymond-ndibe: [jobs-cli] use imagename in definedjob [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T389118) [06:52:34] (03merge) 10raymond-ndibe: [jobs-api] refactor imagename field in models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/149 (https://phabricator.wikimedia.org/T389118) [06:52:38] (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [06:52:38] (03merge) 10raymond-ndibe: [jobs-api] move core logic to seperate core module [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T359808) [06:55:01] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.363-20250407065250-1453d60e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/738 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T389118) [06:59:33] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [07:11:34] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [07:13:22] (03open) 10raymond-ndibe: Revert "[jobs-api] move core logic to seperate core module" [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/152 [07:17:08] (03approved) 10raymond-ndibe: Revert "[jobs-api] move core logic to seperate core module" [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/152 [07:17:12] (03merge) 10raymond-ndibe: Revert "[jobs-api] move core logic to seperate core module" [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/152 [07:19:33] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.363-20250407065250-1453d60e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/738 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T389118) [07:19:36] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.363-20250407065250-1453d60e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/738 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T389118) [07:29:17] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) [07:29:37] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] (split_logic_from_api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [07:29:55] (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] (move_most_custom_validations_out_of_api_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T359804) [07:30:18] (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [07:32:11] (03open) 10raymond-ndibe: [jobs-api] move core logic to seperate core module [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/153 (https://phabricator.wikimedia.org/T359804) [07:32:39] (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (split_logic_from_api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [07:34:25] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [07:45:00] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [07:47:10] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [07:59:45] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [07:59:50] (03update) 10raymond-ndibe: Revert "[jobs-api] move core logic to seperate core module" [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/152 [08:04:56] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.363-20250407065250-1453d60e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/738 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T389118) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:05:04] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.363-20250407065250-1453d60e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/738 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T389118) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:05:10] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.363-20250407065250-1453d60e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/738 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T389118) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:05:31] (03merge) 10raymond-ndibe: [jobs-cli] use imagename in definedjob [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T389118) [08:17:26] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.10 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T389118) [08:18:33] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [08:28:59] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [08:29:43] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [08:41:52] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [08:44:06] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.10 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T389118) [08:44:08] (03approved) 10raymond-ndibe: d/changelog: bump to 16.1.10 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T389118) [08:44:12] (03merge) 10raymond-ndibe: d/changelog: bump to 16.1.10 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T389118) [08:51:57] 06cloud-services-team, 06DBA: m5 master db1228 rebooted itself - https://phabricator.wikimedia.org/T391237 (10Marostegui) 03NEW [08:52:26] 06cloud-services-team, 06DBA: m5 master db1228 rebooted itself - https://phabricator.wikimedia.org/T391237#10716765 (10Marostegui) 05Open→03Resolved a:03Marostegui I am going to close this as fixed, I created it to have a record of this incident/issue. [09:53:30] (03open) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [09:53:33] (03update) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [09:54:28] (03update) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [10:54:04] (03merge) 10pepepiton: T388575 Filter: is author in public domain [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/2 (owner: 10piracalamina) [10:58:06] (03update) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [10:58:23] (03update) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [11:00:27] (03update) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [11:02:56] (03update) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [11:07:15] (03update) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [11:08:38] (03update) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [11:17:26] (03update) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [11:17:56] (03approved) 10fnegri: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) (owner: 10aborrero) [11:20:16] 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10717183 (10Volans) Thanks for the fixes @Andrew, I did another pass, replying to my own comments inline: >>! In T376400#10694816, @Volans wrote: >... [11:23:12] (03merge) 10aborrero: codfw1dev: create network tests instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/170 (https://phabricator.wikimedia.org/T380728) [11:23:16] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:23:48] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [11:24:10] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:24:43] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [11:28:07] (03update) 10aborrero: codfw1dev: flavors: fix project access for g4.cores1.ram1.disk4 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/171 (https://phabricator.wikimedia.org/T380728) [11:28:10] (03open) 10aborrero: codfw1dev: flavors: fix project access for g4.cores1.ram1.disk4 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/171 (https://phabricator.wikimedia.org/T380728) [11:29:14] (03merge) 10aborrero: codfw1dev: flavors: fix project access for g4.cores1.ram1.disk4 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/171 (https://phabricator.wikimedia.org/T380728) [11:29:19] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:29:51] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [11:29:54] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:30:24] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [11:54:28] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 (T390214) [11:54:32] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [11:55:12] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 (T390214) [11:55:25] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214#10717280 (10fnegri) [11:57:00] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 (T390214) [12:02:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [12:07:14] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [12:07:41] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 (T390214) [12:07:45] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:15:36] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 (T390214) [12:15:41] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:20:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [12:22:03] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 (T390214) [12:22:07] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:22:25] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 (T390214) [12:24:09] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/15 [12:25:14] RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [12:26:44] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [12:30:57] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 (T390214) [12:31:01] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:31:13] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: opentofu-created flavors may be disabled by default - https://phabricator.wikimedia.org/T391252 (10aborrero) 03NEW [12:31:44] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [12:35:35] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214#10717375 (10fnegri) [12:36:50] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 (T390214) [12:36:54] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:37:25] !log fnegri@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 (T390214) [12:38:58] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 (T390214) [12:40:14] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 (T390214) [12:40:15] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 (T390214) [12:41:43] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 (T390214) [12:41:44] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 (T390214) [12:41:46] !log fnegri@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 (T390214) [12:41:47] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 (T390214) [12:42:05] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 (T390214) [12:42:09] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:42:52] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 (T390214) [12:42:58] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 (T390214) [12:43:18] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 (T390214) [12:43:19] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 (T390214) [12:43:39] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 (T390214) [12:43:40] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 (T390214) [12:44:25] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 (T390214) [12:44:26] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 (T390214) [12:44:29] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 (T390214) [12:44:30] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 (T390214) [12:44:48] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 (T390214) [12:44:49] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 (T390214) [12:45:18] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 (T390214) [12:45:19] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 (T390214) [12:46:03] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 (T390214) [12:46:04] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 (T390214) [12:46:06] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 (T390214) [12:46:07] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 (T390214) [12:46:15] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 (T390214) [12:46:16] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 (T390214) [12:46:47] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 (T390214) [12:46:48] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 (T390214) [12:47:36] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 (T390214) [12:47:37] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 (T390214) [12:47:40] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:47:40] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 (T390214) [12:47:40] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 (T390214) [12:47:42] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 (T390214) [12:47:42] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 (T390214) [12:48:15] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 (T390214) [12:48:16] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 (T390214) [12:49:03] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 (T390214) [12:49:04] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 (T390214) [12:49:10] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 (T390214) [12:49:11] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 (T390214) [12:49:12] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 (T390214) [12:49:13] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 (T390214) [12:49:45] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 (T390214) [12:49:46] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 (T390214) [12:50:33] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 (T390214) [12:50:35] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 (T390214) [12:50:38] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 (T390214) [12:50:39] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 (T390214) [12:50:42] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 (T390214) [12:50:43] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 (T390214) [12:51:12] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 (T390214) [12:51:13] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 (T390214) [12:52:07] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 (T390214) [12:52:08] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 (T390214) [12:52:09] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 (T390214) [12:52:10] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 (T390214) [12:52:15] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 (T390214) [12:52:16] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 (T390214) [12:52:43] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 (T390214) [12:52:44] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 (T390214) [12:52:47] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:53:40] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 (T390214) [12:53:41] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 (T390214) [12:53:42] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 (T390214) [12:53:43] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 (T390214) [12:53:48] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 (T390214) [12:53:49] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 (T390214) [12:54:19] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 (T390214) [12:54:20] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 (T390214) [12:55:11] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 (T390214) [12:55:12] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 (T390214) [12:55:21] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 (T390214) [12:55:21] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 (T390214) [12:55:22] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 (T390214) [12:55:23] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 (T390214) [12:55:52] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 (T390214) [12:55:53] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 (T390214) [12:56:43] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 (T390214) [12:56:44] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 (T390214) [12:56:48] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 (T390214) [12:56:49] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 (T390214) [12:56:56] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 (T390214) [12:56:57] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 (T390214) [12:57:22] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 (T390214) [12:57:23] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 (T390214) [12:58:16] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 (T390214) [12:58:17] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 (T390214) [12:58:18] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 (T390214) [12:58:19] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [12:58:19] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 (T390214) [12:58:29] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 (T390214) [12:58:30] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 (T390214) [12:58:53] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 (T390214) [12:58:54] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 (T390214) [12:59:46] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 (T390214) [12:59:47] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 (T390214) [12:59:52] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 (T390214) [12:59:53] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 (T390214) [13:00:07] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 (T390214) [13:00:08] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 (T390214) [13:00:28] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 (T390214) [13:00:29] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 (T390214) [13:01:02] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 (T390214) [13:01:03] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 (T390214) [13:01:06] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 (T390214) [13:01:07] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 (T390214) [13:01:43] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 (T390214) [13:01:44] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 (T390214) [13:02:02] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 (T390214) [13:02:03] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 (T390214) [13:02:36] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 (T390214) [13:02:37] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 (T390214) [13:02:38] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 (T390214) [13:02:39] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 (T390214) [13:03:17] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 (T390214) [13:03:17] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 (T390214) [13:03:19] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [13:03:37] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 (T390214) [13:03:38] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 (T390214) [13:04:10] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 (T390214) [13:04:11] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 (T390214) [13:04:12] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 (T390214) [13:04:13] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 (T390214) [13:04:49] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 (T390214) [13:05:08] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 (T390214) [13:05:09] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 (T390214) [13:05:43] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 (T390214) [13:05:43] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 (T390214) [13:05:45] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 (T390214) [13:05:45] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 (T390214) [13:06:38] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 (T390214) [13:07:13] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 (T390214) [13:07:14] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 (T390214) [13:07:17] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 (T390214) [13:08:47] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 (T390214) [13:08:48] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 (T390214) [13:08:51] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [13:10:20] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 (T390214) [13:10:21] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 (T390214) [13:11:48] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 (T390214) [13:15:57] 06cloud-services-team, 10Cloud-VPS, 07IPv6: IPv6 for cloud-realm services - https://phabricator.wikimedia.org/T379282#10717661 (10cmooney) Ok so the IPv6 routing is in place in both eqiad and codfw. I just added the gw IP on the switch for //cloud-private-b1-codfw// a moment ago - 2a02:ec80:a100:205::1/64.... [13:32:23] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 (T390214) [13:32:28] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [13:33:52] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 (T390214) [13:35:33] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214#10717749 (10fnegri) [13:36:34] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 (T390214) [13:40:24] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 (T390214) [13:40:28] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [13:44:46] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 (T390214) [13:46:09] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: IPv6 for cloud-realm services - https://phabricator.wikimedia.org/T379282#10717782 (10cmooney) @taavi also bear in mind for the eqiad side we need an 'aggregate' route similar to this: ` cmooney@cloudservices1005:~$ ip route get fibmatch 172.2... [13:46:11] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 (T390214) [13:46:15] T390214: Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214 [13:47:35] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 (T390214) [13:48:53] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 (T390214) [13:49:07] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214#10717792 (10fnegri) [13:55:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214#10717815 (10fnegri) [13:55:40] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Upgrade "tools" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390214#10717828 (10fnegri) 05In progress→03Resolved All done! [13:56:32] (03open) 10aborrero: codfw1dev: testlabs: use different flavor, and image w/ puppet [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/172 (https://phabricator.wikimedia.org/T391252) [13:57:29] (03merge) 10aborrero: codfw1dev: testlabs: use different flavor, and image w/ puppet [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/172 (https://phabricator.wikimedia.org/T391252) [13:57:30] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:58:03] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [14:02:24] (03update) 10fnegri: Upgrade Kubernetes to 1.29 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 (https://phabricator.wikimedia.org/T362868) [14:03:33] (03open) 10aborrero: instances: use flavor_id instead of name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/173 [14:04:38] (03update) 10aborrero: instances: use flavor_id instead of name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/173 [14:05:06] (03merge) 10aborrero: instances: use flavor_id instead of name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/173 [14:05:09] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:06:24] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:06:40] (03update) 10fnegri: Upgrade Kubernetes to 1.29 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 (https://phabricator.wikimedia.org/T362868) [14:08:34] 06cloud-services-team, 10Cloud-VPS, 07Epic, 13Patch-For-Review: tofu-infra: opentofu-created flavors may be disabled by default - https://phabricator.wikimedia.org/T391252#10717911 (10aborrero) [14:08:36] 06cloud-services-team, 10Cloud-VPS, 07Epic, 13Patch-For-Review: tofu-infra: opentofu-created flavors may be disabled by default - https://phabricator.wikimedia.org/T391252#10717912 (10aborrero) p:05Triage→03Low [14:09:09] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [14:09:36] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [14:33:43] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudvirt10[68-76] - https://phabricator.wikimedia.org/T382492#10717997 (10Jclark-ctr) a:03VRiley-WMF [14:34:01] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudvirt10[68-76] - https://phabricator.wikimedia.org/T382492#10718001 (10Jclark-ctr) @VRiley-WMF procurement T379370 looks like you updated racking ticket Jan 30 2025 have these been Received in coupa / or can the ticke... [14:37:46] 10Striker, 10Bitu, 06Infrastructure-Foundations: Cannot login to toolsadmin, password reset link says invalid token at idm - https://phabricator.wikimedia.org/T391259 (10DaxServer) 03NEW [14:38:33] (03update) 10hoo: Merge cache data from independent test runs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/6 (https://phabricator.wikimedia.org/T388767) (owner: 10arthurtaylor) [14:49:14] 06cloud-services-team, 06DBA: m5 master db1228 rebooted itself - https://phabricator.wikimedia.org/T391237#10718124 (10taavi) 05Resolved→03Open I believe the haproxy reload step from https://wikitech.wikimedia.org/wiki/HAProxy#Failover was missed `lang=shell-session taavi@dbproxy1029 ~ $ echo "show stat" |... [14:49:35] 06cloud-services-team, 06DBA: m5 master db1228 rebooted itself - https://phabricator.wikimedia.org/T391237#10718126 (10taavi) [14:49:39] 10Striker, 10Bitu, 06Infrastructure-Foundations: Cannot login to toolsadmin, password reset link says invalid token at idm - https://phabricator.wikimedia.org/T391259#10718127 (10taavi) [14:52:32] (03approved) 10hoo: Merge cache data from independent test runs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/6 (https://phabricator.wikimedia.org/T388767) (owner: 10arthurtaylor) [14:52:58] (03merge) 10hoo: Merge cache data from independent test runs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/6 (https://phabricator.wikimedia.org/T388767) (owner: 10arthurtaylor) [14:57:47] 10Striker, 10Bitu, 06Infrastructure-Foundations: Cannot login to toolsadmin, password reset link says invalid token at idm - https://phabricator.wikimedia.org/T391259#10718144 (10SLyngshede-WMF) p:05Triage→03High [15:17:59] (03open) 10aborrero: codfw1dev: drop testlabs instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/174 [15:20:15] (03merge) 10aborrero: codfw1dev: drop testlabs instances [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/174 [15:20:31] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:21:14] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:27:10] 06cloud-services-team, 06DBA: m5 master db1228 rebooted itself - https://phabricator.wikimedia.org/T391237#10718305 (10Marostegui) 05Open→03Resolved Fixed it. Thanks for the heads up. [15:32:08] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#10718354 (10LSobanski) p:05Triage→03Low [15:44:11] (03update) 10fnegri: Upgrade Kubernetes to 1.29 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 (https://phabricator.wikimedia.org/T362868) [15:44:56] (03update) 10fnegri: Upgrade Kubernetes to 1.29 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 (https://phabricator.wikimedia.org/T362868) [15:45:49] (03update) 10fnegri: Upgrade Kubernetes to 1.29 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 (https://phabricator.wikimedia.org/T362868) [15:48:18] 06cloud-services-team: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T390906#10718401 (10fnegri) →14Duplicate dup:03T383723 [15:48:20] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10718403 (10fnegri) [15:49:25] 06cloud-services-team: KernelErrors - https://phabricator.wikimedia.org/T390893#10718417 (10fnegri) [15:49:26] 06cloud-services-team: KernelErrors Server cloudcontrol1006 logged kernel errors - https://phabricator.wikimedia.org/T390889#10718415 (10fnegri) →14Duplicate dup:03T390893 [15:49:28] 06cloud-services-team: KernelErrors Server cloudcontrol1005 logged kernel errors - https://phabricator.wikimedia.org/T390886#10718419 (10fnegri) →14Duplicate dup:03T390893 [15:49:29] 06cloud-services-team: KernelErrors - https://phabricator.wikimedia.org/T390893#10718421 (10fnegri) [15:55:02] (03PS1) 10Arendpieter: Switch username validation to Bitu API [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) [15:55:55] 06cloud-services-team: KernelErrors - https://phabricator.wikimedia.org/T390893#10718450 (10fnegri) 05Open→03Resolved a:03fnegri These were triggered during an Openstack upgrade. They've already been purged by journalctl, but I think they were caused by this line I can find in `dmesg`: ` fnegri@cloudc... [15:57:13] (03CR) 10Arendpieter: "This is my first pull request here :) Sorry if I did anything wrong. Could you please review it?" [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) (owner: 10Arendpieter) [16:00:24] 10Striker, 10Bitu, 06Infrastructure-Foundations: Cannot login to toolsadmin, password reset link says invalid token at idm - https://phabricator.wikimedia.org/T391259#10718460 (10SLyngshede-WMF) [16:02:07] 06cloud-services-team, 06DBA: m5 master db1228 rebooted itself - https://phabricator.wikimedia.org/T391237#10718474 (10SLyngshede-WMF) [16:02:09] 10Striker, 10Bitu, 06Infrastructure-Foundations: Cannot login to toolsadmin, password reset link says invalid token at idm - https://phabricator.wikimedia.org/T391259#10718473 (10SLyngshede-WMF) [16:07:25] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10718484 (10fnegri) @VRiley-WMF sorry for the lack of updates, I had to prioritize other things. This server is still alerting, and I didn't manage... [16:19:14] 06cloud-services-team: MetricsinfraAlertmanagerDown Metricsinfra alertmanager is unreachable # page - https://phabricator.wikimedia.org/T390883#10718532 (10fnegri) 05Open→03Resolved a:03fnegri This only fired for about 5 minutes on Apr, 2nd. Maybe some network issue? Resolving for now. {F59006200} [16:20:32] 06cloud-services-team: SystemdUnitDown The systemd unit purge_vm_backup.service on node cloudbackup1004 has been failing for more than two hours. - https://phabricator.wikimedia.org/T390921#10718553 (10fnegri) 05Open→03Resolved a:03fnegri This is not firing anymore. [16:21:31] 06cloud-services-team, 10Toolforge: [jobs-api] Generate the openapi definition from the code - https://phabricator.wikimedia.org/T390138#10718558 (10fnegri) p:05Triage→03Medium [16:29:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance runner-1027 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:30:49] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10718591 (10VRiley-WMF) I'm available for this activity at anytime. 15:00 UTC - 17:00 UTC works for me. [16:32:48] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10718604 (10fnegri) Great! I talked with @VRiley-WMF and the plan is: 1. I'm gonna shut down the server tomorrow for about 1 hour, to check if ther... [16:34:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance runner-1027 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:36:55] 10Striker, 13Patch-For-Review: Add Bitu container to Striker development environment - https://phabricator.wikimedia.org/T362318#10718621 (10Arendpieter) Is this done? There haven’t been any updates for about a year... [16:39:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance runner-1025 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:44:28] FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:49:28] FIRING: [9x] PuppetAgentNoResources: No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:54:28] FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:13:51] 10Tools, 10Wikidata, 07Security: Blocked Wikidata user sockpuppets are doing automated misconduct with QuickStatements - https://phabricator.wikimedia.org/T386978#10718712 (10matej_suchanek) >>! In T386978#10573119, @Epidosis wrote: > By the way, restoring the right for Wikidata administrators to block batch... [17:31:51] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/15 (owner: 10l10n-bot) [17:31:53] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/15 (owner: 10l10n-bot) [17:36:26] (03update) 10raymond-ndibe: deployment: time out stuck deployment after 1h [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/59 (owner: 10dcaro) [17:36:27] (03approved) 10raymond-ndibe: deployment: time out stuck deployment after 1h [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/59 (owner: 10dcaro) [17:38:23] (03update) 10raymond-ndibe: [components-api] update poetry.lock [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/60 [17:38:27] (03update) 10raymond-ndibe: [components-api] update poetry.lock [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/60 [17:38:46] (03update) 10raymond-ndibe: deployment: add run information [repos/cloud/toolforge/components-api] (time_out_deployments) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/61 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [18:08:06] (03approved) 10raymond-ndibe: [components-api] update poetry.lock [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/60 [18:08:11] (03merge) 10raymond-ndibe: [components-api] update poetry.lock [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/60 [18:09:10] (03update) 10raymond-ndibe: deployment: time out stuck deployment after 1h [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/59 (owner: 10dcaro) [18:09:32] (03update) 10raymond-ndibe: deployment: add run information [repos/cloud/toolforge/components-api] (time_out_deployments) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/61 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [18:10:23] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.96-20250407180823-87f2ba60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/739 [18:10:50] (03merge) 10raymond-ndibe: deployment: time out stuck deployment after 1h [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/59 (owner: 10dcaro) [18:10:52] (03update) 10dcaro: deployment: add run information [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/61 (https://phabricator.wikimedia.org/T389339) [18:13:14] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.96-20250407180823-87f2ba60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/739 [18:13:17] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.96-20250407180823-87f2ba60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/739 [18:15:55] (03CR) 10Majavah: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) (owner: 10Arendpieter) [18:18:06] (03CR) 10CI reject: [V:04-1] Switch username validation to Bitu API [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) (owner: 10Arendpieter) [18:24:41] 10Tool-paulina: Filter: is author in public domain - https://phabricator.wikimedia.org/T388575#10718928 (10Piracalamina) 05Open→03Resolved [18:38:17] 10Striker, 10Bitu, 06Infrastructure-Foundations: Cannot login to toolsadmin, password reset link says invalid token at idm - https://phabricator.wikimedia.org/T391259#10718971 (10taavi) Hopefully this is resolved with the database issue subtask being fixed. If not, please re-open. [18:38:23] 06cloud-services-team, 06DBA: m5 master db1228 rebooted itself - https://phabricator.wikimedia.org/T391237#10718973 (10taavi) [18:38:29] 06cloud-services-team, 06DBA: m5 master db1228 rebooted itself - https://phabricator.wikimedia.org/T391237#10718975 (10taavi) [18:38:45] 10Striker, 10Bitu, 06Infrastructure-Foundations: Cannot login to toolsadmin, password reset link says invalid token at idm - https://phabricator.wikimedia.org/T391259#10718976 (10taavi) 05Open→03Resolved [18:38:49] (03update) 10raymond-ndibe: runs: add runs to the commands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/25 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [18:39:15] (03update) 10raymond-ndibe: runs: add runs to the commands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/25 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [18:40:31] (03update) 10raymond-ndibe: deployment: add run information [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/61 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [18:40:34] (03approved) 10raymond-ndibe: deployment: add run information [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/61 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [18:40:41] (03merge) 10raymond-ndibe: deployment: add run information [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/61 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [18:40:42] (03update) 10dcaro: runs: add short message with info on failure/success [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/65 [18:42:48] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.96-20250407180823-87f2ba60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/739 [18:42:50] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.96-20250407180823-87f2ba60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/739 (https://phabricator.wikimedia.org/T389339) [18:43:51] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [18:49:52] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [18:56:18] (03update) 10raymond-ndibe: deploy.build: use the component as the image name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/63 (https://phabricator.wikimedia.org/T388830) (owner: 10dcaro) [18:56:48] (03update) 10raymond-ndibe: deploy.build: use the component as the image name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/63 (https://phabricator.wikimedia.org/T388830) (owner: 10dcaro) [18:58:31] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [19:00:58] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api [19:07:56] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [19:13:15] (03update) 10raymond-ndibe: build: add pending state [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/125 (https://phabricator.wikimedia.org/T389954) (owner: 10dcaro) [19:18:10] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [19:26:05] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [19:26:13] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [19:39:48] (03update) 10raymond-ndibe: components-api: bump to 0.0.96-20250407180823-87f2ba60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/739 (https://phabricator.wikimedia.org/T389339) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:39:51] (03approved) 10raymond-ndibe: components-api: bump to 0.0.96-20250407180823-87f2ba60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/739 (https://phabricator.wikimedia.org/T389339) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:40:27] (03merge) 10raymond-ndibe: components-api: bump to 0.0.96-20250407180823-87f2ba60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/739 (https://phabricator.wikimedia.org/T389339) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [19:41:10] (03update) 10raymond-ndibe: runs: add runs to the commands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/25 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [19:41:17] (03approved) 10raymond-ndibe: runs: add runs to the commands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/25 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [19:41:32] (03merge) 10raymond-ndibe: runs: add runs to the commands [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/25 (https://phabricator.wikimedia.org/T389339) (owner: 10dcaro) [19:44:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-65 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:51:31] 10Tool-paulina: Update Wikidata copyright status as a creator (P7763) in Wikidata for people dead over 100 years ago - https://phabricator.wikimedia.org/T388576#10719186 (10Piracalamina) [20:09:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-65 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:09:41] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#10719204 (10JJMC89) > So mostly this question would be "is it even possible to deliver mail from cloud VPS instances to the outside world" o... [20:18:32] (03update) 10raymond-ndibe: build: add pending state [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/125 (https://phabricator.wikimedia.org/T389954) (owner: 10dcaro) [20:18:51] (03update) 10raymond-ndibe: build: add pending state [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/125 (https://phabricator.wikimedia.org/T389954) (owner: 10dcaro) [20:18:57] (03approved) 10raymond-ndibe: build: add pending state [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/125 (https://phabricator.wikimedia.org/T389954) (owner: 10dcaro) [20:19:02] (03merge) 10raymond-ndibe: build: add pending state [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/125 (https://phabricator.wikimedia.org/T389954) (owner: 10dcaro) [20:21:42] (03open) 10raymond-ndibe: d/changelog: bump to 0.0.6 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/27 [20:23:32] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-cli [20:25:09] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli [20:25:43] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-api: bump to 0.0.184-20250407201912-10dbacc5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/740 (https://phabricator.wikimedia.org/T389954) [20:26:07] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-cli [20:36:14] 10Tool-paulina: Update Wikidata copyright status as a creator (P7763) in Wikidata for people dead over 100 years ago - https://phabricator.wikimedia.org/T388576#10719346 (10Piracalamina) actualizados para autores de: [x] Uruguay [x] argentina [x] colombia [x] chile [x] brasil [ ] mexico [ ] bolivia [ ] costa r... [20:37:39] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli [20:44:48] (03update) 10raymond-ndibe: d/changelog: bump to 0.0.6 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/27 [20:44:51] (03approved) 10raymond-ndibe: d/changelog: bump to 0.0.6 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/27 [22:44:05] (03merge) 10raymond-ndibe: d/changelog: bump to 0.0.6 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/27 [22:55:44] (03update) 10raymond-ndibe: general: use fastapi.status for http codes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/64 (owner: 10dcaro) [22:58:47] (03update) 10raymond-ndibe: deploy.build: use the component as the image name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/63 (https://phabricator.wikimedia.org/T388830) (owner: 10dcaro) [22:58:48] (03approved) 10raymond-ndibe: deploy.build: use the component as the image name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/63 (https://phabricator.wikimedia.org/T388830) (owner: 10dcaro) [22:58:53] (03merge) 10raymond-ndibe: deploy.build: use the component as the image name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/63 (https://phabricator.wikimedia.org/T388830) (owner: 10dcaro) [22:59:28] (03update) 10raymond-ndibe: general: use fastapi.status for http codes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/64 (owner: 10dcaro) [23:00:26] (03update) 10raymond-ndibe: general: use fastapi.status for http codes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/64 (owner: 10dcaro) [23:01:15] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.99-20250407225903-bc59462a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/741 (https://phabricator.wikimedia.org/T388830) [23:01:18] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.99-20250407225903-bc59462a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/741 (https://phabricator.wikimedia.org/T388830) [23:04:01] (03update) 10raymond-ndibe: general: use fastapi.status for http codes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/64 (owner: 10dcaro) [23:04:12] (03approved) 10raymond-ndibe: general: use fastapi.status for http codes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/64 (owner: 10dcaro) [23:04:19] (03merge) 10raymond-ndibe: general: use fastapi.status for http codes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/64 (owner: 10dcaro) [23:04:55] (03update) 10raymond-ndibe: start: check the limit of parallel builds [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/126 (https://phabricator.wikimedia.org/T388706) (owner: 10dcaro) [23:06:17] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.99-20250407225903-bc59462a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/741 (https://phabricator.wikimedia.org/T388830) [23:06:20] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.99-20250407225903-bc59462a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/741 (https://phabricator.wikimedia.org/T388830) [23:18:04] (03update) 10raymond-ndibe: runs: add short message with info on failure/success [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/65 (owner: 10dcaro) [23:39:49] (03open) 10chuckonwumelu: Draft: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [23:45:04] 10cloud-services-team (Hardware): cloudcontrol1011 service implementation tracking - https://phabricator.wikimedia.org/T391300 (10RobH) 03NEW p:05Triage→03Medium [23:45:50] 10cloud-services-team (Hardware): cloudcontrol1011 service implementation tracking - https://phabricator.wikimedia.org/T391300#10719732 (10RobH) [23:45:50] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudcontrol1011 - https://phabricator.wikimedia.org/T380499#10719731 (10RobH) [23:52:34] (03update) 10raymond-ndibe: start: check the limit of parallel builds [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/126 (https://phabricator.wikimedia.org/T388706) (owner: 10dcaro) [23:52:44] (03approved) 10raymond-ndibe: start: check the limit of parallel builds [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/126 (https://phabricator.wikimedia.org/T388706) (owner: 10dcaro) [23:52:49] (03merge) 10raymond-ndibe: start: check the limit of parallel builds [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/126 (https://phabricator.wikimedia.org/T388706) (owner: 10dcaro) [23:56:12] (03approved) 10raymond-ndibe: runs: add short message with info on failure/success [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/65 (owner: 10dcaro) [23:56:55] (03merge) 10raymond-ndibe: runs: add short message with info on failure/success [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/65 (owner: 10dcaro) [23:56:57] (03update) 10dcaro: run: mark as skipped if the deploy failed [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/66 [23:58:52] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-api: bump to 0.0.184-20250407201912-10dbacc5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/740 (https://phabricator.wikimedia.org/T389954) [23:58:55] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-api: bump to 0.0.184-20250407201912-10dbacc5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/740 (https://phabricator.wikimedia.org/T388706 https://phabricator.wikimedia.org/T389954) [23:58:55] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.99-20250407225903-bc59462a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/741 (https://phabricator.wikimedia.org/T388830) [23:58:59] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.99-20250407225903-bc59462a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/741 (https://phabricator.wikimedia.org/T388830)