[00:26:17] (03open) 10raymond-ndibe: [toolforge,tool_handlers] replace destination_name comparisons with image_name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/115 (https://phabricator.wikimedia.org/T395076) [00:27:33] 10Toolforge (Toolforge iteration 22), 07good first task, 13Patch-For-Review: [components-api] use the `build.params.image_name` to compare with the `component` - https://phabricator.wikimedia.org/T395076#11059472 (10Raymond_Ndibe) a:03Raymond_Ndibe [00:27:53] 10Toolforge (Toolforge iteration 22), 07good first task, 13Patch-For-Review: [components-api] use the `build.params.image_name` to compare with the `component` - https://phabricator.wikimedia.org/T395076#11059474 (10Raymond_Ndibe) 05Open→03In progress [00:40:18] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/49 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:40:23] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/49 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:40:30] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/49 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:40:47] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/51 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:40:47] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/51 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:40:50] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/51 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:40:56] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/259 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:40:57] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/259 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:00] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/259 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:05] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:06] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:10] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:25] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/48 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:31] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/48 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:44] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/138 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:44] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/138 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:48] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/138 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:54] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/61 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:55] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/61 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:41:58] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/61 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:42:07] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:42:16] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:42:17] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:42:37] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/88 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:42:38] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/88 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:42:48] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/88 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:43:16] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/32 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:43:21] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/32 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:43:24] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/87 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:43:26] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/87 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:43:30] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/87 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:43:38] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/190 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:43:39] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/190 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:43:51] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/190 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:43:54] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/48 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:03] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.58-20250805004101-e6674da0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/893 [00:44:11] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.58-20250805004101-e6674da0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/893 [00:44:16] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/24 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:19] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/24 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:25] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/116 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:27] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/24 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:30] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/116 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:41] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/28 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:43] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/116 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:49] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/28 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:49] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.74-20250805004122-c72ec4f3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/894 [00:44:50] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.74-20250805004122-c72ec4f3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/894 [00:44:54] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/114 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:59] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/28 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:59] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/114 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:07] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/191 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:09] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/114 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:13] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/191 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:27] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/191 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:33] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/191 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:37] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/26 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:41] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/26 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:49] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:51] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/26 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:45:53] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:46:14] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:46:18] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/27 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:46:34] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/87 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:46:35] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/27 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:46:40] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/27 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:03] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/115 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:10] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:12] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/115 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:13] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/115 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:24] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.74-20250805004122-c72ec4f3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/894 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:25] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.74-20250805004122-c72ec4f3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/894 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:28] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.74-20250805004122-c72ec4f3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/894 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:40] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.58-20250805004101-e6674da0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/893 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:45] (03approved) 10raymond-ndibe: maintain-harbor: bump to 0.0.58-20250805004101-e6674da0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/893 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:47:46] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.58-20250805004101-e6674da0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/893 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:48:10] (03merge) 10raymond-ndibe: maintain-harbor: bump to 0.0.58-20250805004101-e6674da0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/893 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:48:18] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/190 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:48:27] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.69-20250805004333-2e8404df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/895 [00:48:38] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.69-20250805004333-2e8404df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/895 [00:48:42] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.63-20250805004518-a882bb63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/896 [00:48:43] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.63-20250805004518-a882bb63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/896 [00:48:59] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/115 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:50:04] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.63-20250805004518-a882bb63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/896 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:50:05] (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.63-20250805004518-a882bb63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/896 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:50:11] (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.63-20250805004518-a882bb63 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/896 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:50:25] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.75-20250805004735-9b30c822 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/897 [00:50:25] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.75-20250805004735-9b30c822 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/897 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:50:26] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.75-20250805004735-9b30c822 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/897 [00:50:34] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.75-20250805004735-9b30c822 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/897 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:50:47] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.62-20250805004635-88bcb680 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/898 [00:50:50] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.62-20250805004635-88bcb680 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/898 [00:51:07] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.138-20250805004550-cbb3939c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/899 [00:51:07] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.138-20250805004550-cbb3939c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/899 [00:51:24] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.31-20250805004511-3ae60417 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/900 [00:51:26] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.63-20250805004617-1686e33f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/901 [00:51:27] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.31-20250805004511-3ae60417 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/900 [00:51:33] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.63-20250805004617-1686e33f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/901 [00:51:39] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.75-20250805004735-9b30c822 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/897 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:51:53] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.391-20250805004900-66c5cfb5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/902 [00:51:54] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/191 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:51:57] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.391-20250805004900-66c5cfb5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/902 [00:52:08] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.72-20250805004241-857f13a2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/903 [00:52:14] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.72-20250805004241-857f13a2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/903 [00:52:16] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/27 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:53:10] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.196-20250805004201-0005e421 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/904 [00:54:43] (03update) 10raymond-ndibe: volume-admission: bump to 0.0.69-20250805004333-2e8404df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/895 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:54:44] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/191 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:55:18] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [00:56:46] (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/56 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:56:47] (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/56 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:56:49] (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/56 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:56:55] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/113 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:56:57] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/113 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:56:59] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/113 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:57:07] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/118 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:57:09] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/118 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:57:14] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/118 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:57:18] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.392-20250805005500-28b59168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/902 [00:57:21] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.392-20250805005500-28b59168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/902 [00:57:27] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/55 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:57:31] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/55 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:57:32] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/55 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:58:29] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [00:58:35] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [00:58:41] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/113 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:58:53] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/55 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:59:33] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [01:02:27] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [01:02:41] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [01:04:33] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [01:09:57] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [01:10:09] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [01:10:19] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [01:16:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [01:18:12] (03update) 10raymond-ndibe: volume-admission: bump to 0.0.69-20250805004333-2e8404df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/895 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:18:13] (03approved) 10raymond-ndibe: volume-admission: bump to 0.0.69-20250805004333-2e8404df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/895 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:18:16] (03merge) 10raymond-ndibe: volume-admission: bump to 0.0.69-20250805004333-2e8404df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/895 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:18:33] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.392-20250805005500-28b59168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/902 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:18:34] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.392-20250805005500-28b59168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/902 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:18:35] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.392-20250805005500-28b59168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/902 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:19:08] (03update) 10raymond-ndibe: builds-api: bump to 0.0.196-20250805004201-0005e421 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/904 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:19:22] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [01:20:04] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.392-20250805005500-28b59168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/902 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:21:18] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [01:21:22] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [01:23:08] (03open) 10raymond-ndibe: d/changelog: bump to 0.0.23 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/117 [01:24:15] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [01:24:55] (03open) 10raymond-ndibe: d/changelog: bump to 0.0.14 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/57 [01:25:58] (03open) 10raymond-ndibe: d/changelog: bump to 0.3.7 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/50 [01:26:34] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.17 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/119 [01:29:19] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [01:30:03] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [01:32:44] 10Toolforge (Toolforge iteration 22), 07good first task, 13Patch-For-Review: [components-api] use the `build.params.image_name` to compare with the `component` - https://phabricator.wikimedia.org/T395076#11059504 (10Raymond_Ndibe) 05In progress→03Resolved [01:32:51] 10Toolforge (Toolforge iteration 22), 07good first task, 13Patch-For-Review: [components-api] use the `build.params.image_name` to compare with the `component` - https://phabricator.wikimedia.org/T395076#11059507 (10Raymond_Ndibe) 05Resolved→03In progress [01:37:04] (03update) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) [01:37:17] (03approved) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) [01:37:20] (03update) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) [01:44:30] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [01:44:57] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [01:45:24] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [01:47:23] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [01:48:55] (03update) 10raymond-ndibe: builds-api: bump to 0.0.196-20250805004201-0005e421 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/904 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:49:11] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.72-20250805004241-857f13a2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/903 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:49:47] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [01:50:58] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [01:52:48] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.63-20250805004617-1686e33f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/901 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [01:53:03] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [01:59:27] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [01:59:55] (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.31-20250805004511-3ae60417 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/900 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:00:13] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [02:02:29] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [02:04:01] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [02:06:56] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [02:08:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [02:14:41] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [02:16:09] FIRING: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [02:17:10] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [02:17:39] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [02:19:10] (03update) 10raymond-ndibe: components-api: bump to 0.0.138-20250805004550-cbb3939c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/899 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:19:27] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [02:21:26] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [02:22:41] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [02:22:47] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [02:23:29] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [02:27:31] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [02:29:49] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [02:30:22] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [02:30:25] !log raymond-ndibe@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor [02:30:33] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [02:31:07] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.62-20250805004635-88bcb680 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/898 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:31:33] (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.31-20250805004511-3ae60417 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/900 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:31:34] (03approved) 10raymond-ndibe: envvars-admission: bump to 0.0.31-20250805004511-3ae60417 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/900 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:31:37] (03merge) 10raymond-ndibe: envvars-admission: bump to 0.0.31-20250805004511-3ae60417 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/900 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:32:12] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.63-20250805004617-1686e33f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/901 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:33:20] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [02:33:43] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [02:34:48] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [02:34:51] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [02:36:09] RESOLVED: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [02:36:50] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [02:36:54] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [02:39:42] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.62-20250805004635-88bcb680 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/898 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:39:57] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [02:40:25] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [02:41:23] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.72-20250805004241-857f13a2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/903 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:41:30] (03update) 10raymond-ndibe: builds-api: bump to 0.0.196-20250805004201-0005e421 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/904 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:41:51] (03update) 10raymond-ndibe: components-api: bump to 0.0.138-20250805004550-cbb3939c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/899 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:47:32] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [02:49:55] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.63-20250805004617-1686e33f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/901 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:49:56] (03approved) 10raymond-ndibe: registry-admission: bump to 0.0.63-20250805004617-1686e33f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/901 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:50:07] (03merge) 10raymond-ndibe: registry-admission: bump to 0.0.63-20250805004617-1686e33f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/901 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:50:27] (03update) 10raymond-ndibe: components-api: bump to 0.0.138-20250805004550-cbb3939c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/899 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:50:47] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [02:55:27] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [02:57:22] (03update) 10raymond-ndibe: components-api: bump to 0.0.138-20250805004550-cbb3939c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/899 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:57:22] (03approved) 10raymond-ndibe: components-api: bump to 0.0.138-20250805004550-cbb3939c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/899 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:57:28] (03merge) 10raymond-ndibe: components-api: bump to 0.0.138-20250805004550-cbb3939c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/899 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:57:41] (03update) 10raymond-ndibe: builds-api: bump to 0.0.196-20250805004201-0005e421 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/904 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:57:44] (03approved) 10raymond-ndibe: builds-api: bump to 0.0.196-20250805004201-0005e421 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/904 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:57:46] (03update) 10raymond-ndibe: builds-api: bump to 0.0.196-20250805004201-0005e421 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/904 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [02:57:59] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [03:03:45] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [03:04:07] (03merge) 10raymond-ndibe: builds-api: bump to 0.0.196-20250805004201-0005e421 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/904 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:04:13] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.72-20250805004241-857f13a2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/903 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:04:24] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [03:06:05] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [03:07:33] (03approved) 10raymond-ndibe: envvars-api: bump to 0.0.72-20250805004241-857f13a2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/903 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:07:43] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.72-20250805004241-857f13a2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/903 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:07:45] (03merge) 10raymond-ndibe: envvars-api: bump to 0.0.72-20250805004241-857f13a2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/903 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:07:57] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [03:11:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [03:16:56] FIRING: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:19:55] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [03:20:41] (03approved) 10raymond-ndibe: ingress-admission: bump to 0.0.62-20250805004635-88bcb680 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/898 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:20:45] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.62-20250805004635-88bcb680 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/898 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:20:48] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.62-20250805004635-88bcb680 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/898 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:21:02] (03merge) 10raymond-ndibe: ingress-admission: bump to 0.0.62-20250805004635-88bcb680 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/898 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [05:11:56] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:12:11] 06cloud-services-team: SystemdUnitDown The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://phabricator.wikimedia.org/T401161 (10phaultfinder) 03NEW [05:49:55] PROBLEM - toolschecker: NFS read/writeable on labs instances on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/nfs/home - 324 bytes in 60.022 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [05:51:53] RECOVERY - toolschecker: NFS read/writeable on labs instances on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 54.838 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [06:37:57] (03CR) 10Abijeet Patro: "recheck" [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1175500 (owner: 10L10n-bot) [06:59:28] FIRING: InstanceDown: Project tools instance tools-prometheus-9 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:03:05] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-24 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:10:26] (03update) 10vriaa: Draft: Basic banner implementation [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/1 [07:37:12] (03update) 10dcaro: [config] allow reading from stdin [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/54 (https://phabricator.wikimedia.org/T398424) (owner: 10raymond-ndibe) [07:44:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-9 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:56:09] (03update) 10dcaro: api: allow protocol to be specified for ports [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/186 [08:03:05] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-24 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [08:14:13] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11059811 (10dcaro) >>! In T401075#11058519, @DamianZaremba wrote: > I wasn't really planning on doing this today... but after some playing around ht... [08:16:44] 06cloud-services-team, 10Toolforge (Toolforge iteration 22): [builds-builder] review and potentially add cmake buildpack - https://phabricator.wikimedia.org/T401169 (10dcaro) 03NEW [08:18:07] 06cloud-services-team, 10Toolforge: [builds-builder,apt] migrate from apt buildpack to Heroku's .deb packages buildpack - https://phabricator.wikimedia.org/T387141#11059826 (10dcaro) Note that a newer version of the upstream deb packages buildpack allows specifying external repos, that feature should be gated... [08:21:13] 06cloud-services-team, 10Toolforge: Loki usage - https://phabricator.wikimedia.org/T401151#11059833 (10dcaro) The task implementing that is {T400917}, leaving this task open for the volume request info. [08:25:58] 06cloud-services-team, 10Toolforge: [jobs-api] report status pending until the first container is running - https://phabricator.wikimedia.org/T401069#11059836 (10dcaro) [08:34:56] (03update) 10taavi: api: Allow querying logs for non-existent jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/189 (https://phabricator.wikimedia.org/T400913) [08:36:20] 06cloud-services-team, 10Toolforge: [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172 (10dcaro) 03NEW [08:37:14] 06cloud-services-team, 10Toolforge: [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172#11059886 (10dcaro) Given the differences is the states, this might be better after {T390136} [08:37:47] (03merge) 10taavi: api: Allow querying logs for non-existent jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/189 (https://phabricator.wikimedia.org/T400913) [08:40:17] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.393-20250805083756-7ab43158 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/905 (https://phabricator.wikimedia.org/T400913) [08:40:53] (03update) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113) [08:41:25] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:43:46] 06cloud-services-team, 10Toolforge: [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172#11059896 (10dcaro) [08:50:26] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [08:50:42] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:01:13] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:04:15] (03merge) 10taavi: jobs-api: bump to 0.0.393-20250805083756-7ab43158 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/905 (https://phabricator.wikimedia.org/T400913) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:04:56] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] Allow querying logs for non-existent jobs - https://phabricator.wikimedia.org/T400913#11059950 (10taavi) 05Open→03Resolved [09:08:28] 06cloud-services-team, 10Toolforge: [toolforge,jobs] "toolforge jobs logs" fails when job has not started yet - https://phabricator.wikimedia.org/T349775#11059957 (10taavi) 05Open→03Resolved a:05Raymond_Ndibe→03taavi {T398645} should have fixed this if it wasn't fixed some other way already. [09:12:11] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:24:13] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge-static gives 301 redirects to backend server with port 8000 - https://phabricator.wikimedia.org/T401024#11060018 (10taavi) 05Open→03Resolved [09:42:12] (03approved) 10dcaro: [config] allow reading from stdin [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/54 (https://phabricator.wikimedia.org/T398424) (owner: 10raymond-ndibe) [09:42:36] (03update) 10dcaro: [config] allow reading from stdin [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/54 (https://phabricator.wikimedia.org/T398424) (owner: 10raymond-ndibe) [09:42:56] (03update) 10dcaro: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064) (owner: 10raymond-ndibe) [09:45:37] (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe) [09:46:11] (03PS1) 10Gopavasanth: Added depict search url to configuration file [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1175853 (https://phabricator.wikimedia.org/T389122) [09:47:52] (03unapproved) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe) [10:04:28] 06cloud-services-team, 10Toolforge: Support `--timeout` for one-off jobs - https://phabricator.wikimedia.org/T401110#11060152 (10dcaro) p:05Triage→03Medium [10:41:01] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T401133#11060280 (10fnegri) 05Open→03Resolved a:03fnegri Fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1175530 [10:41:10] 06cloud-services-team: PuppetFailure Puppet has failed on clouddumps1001:9100 - https://phabricator.wikimedia.org/T401130#11060283 (10fnegri) Fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1175530 [10:44:53] (03PS1) 10Gopavasanth: Enhanced layout structure with flexbox for better footer positioning [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1175859 (https://phabricator.wikimedia.org/T365438) [10:54:18] 06cloud-services-team: PuppetFailure Puppet has failed on clouddumps1001:9100 - https://phabricator.wikimedia.org/T401130#11060318 (10taavi) 05In progress→03Resolved [10:58:44] 06cloud-services-team, 10Toolforge, 07patch-welcome: toolforge-weld: Outdated license classifier - https://phabricator.wikimedia.org/T401188 (10taavi) 03NEW [10:58:53] 06cloud-services-team, 10Toolforge, 07patch-welcome: toolforge-weld: Outdated license classifier - https://phabricator.wikimedia.org/T401188#11060338 (10taavi) p:05Triage→03Low [11:15:36] 06cloud-services-team, 10Toolforge: [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172#11060417 (10dcaro) p:05Triage→03High [11:16:21] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 [11:19:18] (03open) 10taavi: utils: Provide asyncio version of peek() [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/84 [11:19:22] (03update) 10taavi: utils: Provide asyncio version of peek() [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/84 [11:25:00] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 [11:26:12] 06cloud-services-team, 10Toolforge: [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172#11060429 (10dcaro) [11:32:42] 06cloud-services-team, 10Toolforge: [jobs-api] Support following logs from Loki - https://phabricator.wikimedia.org/T400916#11060439 (10taavi) a:03taavi [11:32:57] 10Toolforge (Toolforge iteration 22): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#11060443 (10dcaro) Adding a small update to this, currently toolforge clis running inside toolforge images (buildservice and pre-built) are now ab... [11:34:54] (03open) 10taavi: Draft: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916) [11:38:51] (03update) 10taavi: Draft: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916) [11:39:59] (03update) 10taavi: utils: Provide asyncio version of peek() [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/84 [11:40:54] (03open) 10dcaro: functional_tests: trigger webservice-cli tests for tool-webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/906 [11:41:29] (03update) 10dcaro: functional_tests: trigger webservice-cli tests for tool-webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/906 [11:52:45] 06cloud-services-team, 10Toolforge: Investigate daily disconnections of IRC bots hosted in Toolforge - https://phabricator.wikimedia.org/T400223#11060473 (10dcaro) I see in the logs from yesterday: ` Aug 04 15:14:31 tools-k8s-worker-nfs-33 kubelet[862]: I0804 15:14:31.578144 862 scope.go:117] "RemoveContai... [12:03:05] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-24 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [12:04:55] (03approved) 10taavi: cli: only send fields that are set [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/112 (owner: 10dcaro) [12:05:08] (03update) 10taavi: Draft: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916) [12:07:23] 06cloud-services-team, 10Toolforge: Investigate daily disconnections of IRC bots hosted in Toolforge - https://phabricator.wikimedia.org/T400223#11060499 (10dcaro) For some reason it's not sending (or failing to) the T1/T2 dhcp requests, and ends up falling to the lifetime: ` root@tools-k8s-worker-nfs-33:~# ca... [12:10:00] 06Toolforge-standards-committee: Adoption request for pagelister - https://phabricator.wikimedia.org/T398111#11060526 (10LucasWerkmeister) 05Open→03Resolved a:03MusikAnimal Alright, then let’s close this :) thanks all! [12:11:43] 06cloud-services-team, 10Toolforge: Investigate daily disconnections of IRC bots hosted in Toolforge - https://phabricator.wikimedia.org/T400223#11060531 (10taavi) DHCP server logs: P80804 The "lease not found" errors do seem a bit concerning. [12:15:59] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [12:16:00] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [12:16:33] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [12:16:37] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli [12:16:54] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice [12:16:57] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice [12:17:09] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tool-webservice [12:17:12] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tool-webservice [12:17:16] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [12:17:17] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [12:18:05] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-24 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [12:19:26] (03update) 10taavi: Draft: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916) [12:19:27] (03update) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113) [12:19:58] (03PS1) 10David Caro: deploy: rename tools-webservice -> webservice-cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1175885 [12:21:14] (03open) 10dcaro: rename webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/907 [12:21:29] (03update) 10dcaro: rename webservice [repos/cloud/toolforge/toolforge-deploy] (fix_webservice_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/907 [12:21:44] (03update) 10dcaro: rename webservice [repos/cloud/toolforge/toolforge-deploy] (fix_webservice_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/907 [12:21:55] (03update) 10dcaro: rename webservice [repos/cloud/toolforge/toolforge-deploy] (fix_webservice_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/907 [12:25:33] (03update) 10dcaro: components: rename webservice [repos/cloud/toolforge/toolforge-deploy] (fix_webservice_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/907 [12:30:22] (03update) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2 [12:31:35] (03update) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2 [12:31:55] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [12:31:57] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli [12:35:24] (03update) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2 [13:12:11] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:32:57] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [toolforge-deploy.tests] account for warning messages printed to stderr - https://phabricator.wikimedia.org/T400390#11060745 (10dcaro) p:05Triage→03Medium [13:37:41] (03merge) 10dcaro: functional_tests: trigger webservice-cli tests for tool-webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/906 [13:37:44] (03update) 10dcaro: components: rename webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/907 [13:37:49] (03approved) 10dcaro: components: rename webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/907 [13:38:33] (03merge) 10dcaro: components: rename webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/907 [13:43:43] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [13:43:44] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli [13:46:22] (03open) 10dcaro: bump_version: use the new webservice-cli component name [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/81 [13:46:31] (03approved) 10dcaro: bump_version: use the new webservice-cli component name [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/81 [13:47:34] (03merge) 10dcaro: bump_version: use the new webservice-cli component name [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/81 [13:49:10] (03open) 10dcaro: d/changelog: bump to 0.103.16 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/82 (https://phabricator.wikimedia.org/T360488 https://phabricator.wikimedia.org/T384788) [13:49:20] (03close) 10dcaro: d/changelog: bump to 0.103.16 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/77 (https://phabricator.wikimedia.org/T360488 https://phabricator.wikimedia.org/T384788) [13:50:05] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [13:54:41] (03open) 10dcaro: functional_tests: run only webservice tests when it is deployed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/908 [13:55:53] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [13:57:38] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [14:01:56] RESOLVED: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:03:19] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [14:22:17] (03approved) 10anzx: Add new i18n file - Kannada(kn) language [toolforge-repos/wstranclude] - 10https://gitlab.wikimedia.org/toolforge-repos/wstranclude/-/merge_requests/1 [14:22:26] (03update) 10raymond-ndibe: [toolforge,tool_handlers] replace destination_name comparisons with image_name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/115 (https://phabricator.wikimedia.org/T395076) [14:31:31] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [14:36:12] 06cloud-services-team, 10Toolforge: [jobs-api] Allow customizing time to request Loki logs for - https://phabricator.wikimedia.org/T400917#11061039 (10DamianZaremba) From my usage of logs it would be useful (on the basis of a time window being explicitly required): 1. `--since N` accepting something like [15m,... [14:37:46] 10Tool-community-activity-alerts-system, 03Wikimania-Hackathon-2025: Community Alerts Dashboard - Migrate to Codex and Improve UI - https://phabricator.wikimedia.org/T401036#11061052 (10Dagmawi-M) a:03Dagmawi-M [14:40:57] 06cloud-services-team, 10Toolforge: [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172#11061072 (10DamianZaremba) Thanks for outlining this - it would be a super useful addition. [14:43:15] (03merge) 10galrach600: Eliza http errors [toolforge-repos/miss-search] (update-cycle-toolforge-testing) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/11 (owner: 10eliza189) [15:05:36] (03CR) 10David Caro: [C:03+2] "Tested with the latest webservice release" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1175885 (owner: 10David Caro) [15:09:37] (03Merged) 10jenkins-bot: deploy: rename tools-webservice -> webservice-cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1175885 (owner: 10David Caro) [15:15:38] 06cloud-services-team, 10Toolforge: Investigate daily disconnections of IRC bots hosted in Toolforge - https://phabricator.wikimedia.org/T400223#11061251 (10dcaro) It seems that the lease is created in one server, but the requests are received in the other, on cloudnet1005, it gave the lease eventually: ` root... [15:16:14] (03approved) 10dcaro: d/changelog: bump to 0.103.16 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/82 (https://phabricator.wikimedia.org/T360488 https://phabricator.wikimedia.org/T384788) [15:16:19] (03merge) 10dcaro: d/changelog: bump to 0.103.16 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/82 (https://phabricator.wikimedia.org/T360488 https://phabricator.wikimedia.org/T384788) [15:36:23] (03open) 10dcaro: tool: add stern for log tailing and such [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/260 [15:36:29] (03update) 10dcaro: tool: add stern for log tailing and such [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/260 [15:40:57] (03update) 10dcaro: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064) (owner: 10raymond-ndibe) [15:44:51] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [kyverno] Upgrade to `3.3.9` chart (`1.13` app) for k8s 1.30 support - https://phabricator.wikimedia.org/T394787#11061431 (10dcaro) [15:44:59] 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [components-api] store the config used for the deployment in the deployment themselves - https://phabricator.wikimedia.org/T400064#11061435 (10dcaro) [15:45:01] 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [jobs-cli,builds-cli,toolforge-cli,components-cli,envvars-cli] move the packaging scripts to bookworm - https://phabricator.wikimedia.org/T400616#11061437 (10dcaro) [15:45:03] 10Toolforge (Toolforge iteration 23), 07good first task, 13Patch-For-Review: [components-api] use the `build.params.image_name` to compare with the `component` - https://phabricator.wikimedia.org/T395076#11061433 (10dcaro) [15:45:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [harbor,infra] Find a way to manage toolforge project policies with code - https://phabricator.wikimedia.org/T360509#11061439 (10dcaro) [15:45:08] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [components-api,beta] CI pipelines should wait until Toolforge deployment is 100% successful - https://phabricator.wikimedia.org/T398485#11061443 (10dcaro) [15:45:11] 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [jobs-api] check for diff in services when running diff_with_running_job - https://phabricator.wikimedia.org/T392717#11061441 (10dcaro) [15:45:17] 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [jobs-api] refactor models - https://phabricator.wikimedia.org/T389118#11061447 (10dcaro) [15:45:20] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [jobs-api] Periodically refresh image-config data - https://phabricator.wikimedia.org/T357112#11061445 (10dcaro) [15:45:25] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [jobs-api] Split the `*Job` API models into three - https://phabricator.wikimedia.org/T390136#11061450 (10dcaro) [15:45:37] 10Toolforge (Toolforge iteration 23), 07Upstream: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#11061469 (10dcaro) [15:45:38] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 07Epic: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#11061466 (10dcaro) [15:45:40] 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11061465 (10dcaro) [15:45:43] 10Toolforge (Toolforge iteration 23), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#11061471 (10dcaro) [15:45:44] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 23), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#11061468 (10dcaro) [15:45:47] 10Toolforge (Toolforge iteration 23): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#11061472 (10dcaro) [15:46:20] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 23), 05Goal: [harbor] Move harbor data to object storage service - https://phabricator.wikimedia.org/T350687#11061483 (10dcaro) [15:46:23] 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [toolforge-deploy.tests] account for warning messages printed to stderr - https://phabricator.wikimedia.org/T400390#11061479 (10dcaro) [15:46:24] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): Support for UDP ports in jobs - https://phabricator.wikimedia.org/T400024#11061485 (10dcaro) [15:46:25] 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [components-cli] Allow reading tool configuration from stdin - https://phabricator.wikimedia.org/T398424#11061481 (10dcaro) [15:46:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [jobs-api] when running a command with wrong quoting, no logs nor useful feedback is given to the user - https://phabricator.wikimedia.org/T356267#11061489 (10dcaro) [15:46:34] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11061487 (10dcaro) [15:46:35] 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: Persist important toolforge k8s components logs - https://phabricator.wikimedia.org/T383081#11061491 (10dcaro) [15:46:37] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11061493 (10dcaro) [15:46:42] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 23), 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#11061497 (10dcaro) [15:46:43] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [builds-builder] Upgrade python buildpack to v0.17.0 or newer for Poetry support - https://phabricator.wikimedia.org/T374056#11061499 (10dcaro) [15:46:45] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: Toolforge: Replace all bastion with grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#11061495 (10dcaro) [15:46:48] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#11061501 (10dcaro) [15:46:52] 10Toolforge (Toolforge iteration 23): [clis] standardize the package names - https://phabricator.wikimedia.org/T399080#11061505 (10dcaro) [15:46:56] 10Cloud Services Proposals, 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 23), 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-d... - https://phabricator.wikimedia.org/T194332#11061503 [15:47:00] 10Toolforge (Toolforge iteration 23): [foxtrot-ldap] publish image in harbor repos - https://phabricator.wikimedia.org/T400167#11061509 (10dcaro) [15:47:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [builds-builder] review and potentially add cmake buildpack - https://phabricator.wikimedia.org/T401169#11061508 (10dcaro) [15:47:08] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge (Toolforge iteration 23): Decision request - Reuse toolforge user tools central logging for toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#11061510 (10dcaro) [15:47:14] 10Toolforge (Toolforge iteration 23): [docs] enable docs linter in one of the repos - https://phabricator.wikimedia.org/T397949#11061512 (10dcaro) [15:47:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#11061511 (10dcaro) [15:47:22] 10Toolforge (Toolforge iteration 23): [builds-cli] add resolved reference when showing a build - https://phabricator.wikimedia.org/T394300#11061514 (10dcaro) [15:47:26] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [tools-static,infra] NFS issues should not bring tools-static down - https://phabricator.wikimedia.org/T397634#11061513 (10dcaro) [15:47:30] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [components-api] optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#11061515 (10dcaro) [15:47:34] 10cloud-services-team (FY2025/26-Q1), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 23), 07Epic, 05Goal: [infra] Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#11061516 (10dcaro) [15:47:50] (03CR) 10Eugene233: "Thank you for your fix. I have some comments you want to look at 😊" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1175859 (https://phabricator.wikimedia.org/T365438) (owner: 10Gopavasanth) [15:48:48] (03approved) 10fnegri: tool: add stern for log tailing and such [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/260 (owner: 10dcaro) [15:52:49] (03merge) 10dcaro: tool: add stern for log tailing and such [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/260 [15:57:33] (03update) 10dcaro: cli: only send fields that are set [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/112 [15:58:43] (03merge) 10dcaro: cli: only send fields that are set [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/112 [16:17:50] (03close) 10raymond-ndibe: d/changelog: bump to 16.1.17 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/119 [16:18:24] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.17 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/120 [16:24:05] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [16:27:55] (03update) 10vriaa: Basic banner implementation [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/1 [16:29:16] (03open) 10vriaa: feat: Add NotFound page for unknown routes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/2 [16:29:21] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [16:30:12] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [16:31:27] (03open) 10vriaa: docs: Update contributing guide for developers without repo access [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/3 [16:36:59] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [16:37:32] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.17 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/120 [16:43:51] (03approved) 10raymond-ndibe: d/changelog: bump to 16.1.17 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/120 [16:43:55] (03merge) 10raymond-ndibe: d/changelog: bump to 16.1.17 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/120 [16:56:11] (03update) 10raymond-ndibe: [cli] Change port type to allow protocol suffix [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/115 (https://phabricator.wikimedia.org/T400024) [17:06:01] (03update) 10raymond-ndibe: api: allow protocol to be specified for ports [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/186 (owner: 10dcaro) [17:11:12] 06cloud-services-team, 10Cloud-VPS: Request to enable XFF headers for test XTools hostnames - https://phabricator.wikimedia.org/T400964#11061891 (10MusikAnimal) I was expecting this to be an uncontroversial request that would be actioned quickly. If there's any hesitation, we can just decline this. I put a lot... [17:15:08] 10Tool-wikicordo: Limit input on Tool-wikicordo resets to 100 when set below 100 - https://phabricator.wikimedia.org/T401232 (10Josve05a) 03NEW [17:43:00] 06cloud-services-team, 10Toolforge: What is the preferred method of exporting metrics from jobs? - https://phabricator.wikimedia.org/T401235 (10DamianZaremba) 03NEW [17:45:29] 06cloud-services-team, 10Toolforge, 07Epic: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools - https://phabricator.wikimedia.org/T127367#11062033 (10dcaro) [17:46:14] 06cloud-services-team, 10Toolforge, 07Epic: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools - https://phabricator.wikimedia.org/T127367#11062045 (10dcaro) Restoring the unified logging service description, that should not be lost, and should be tackled soon ri... [17:49:12] 06cloud-services-team, 10Toolforge: toolforge jobs logs throws an exception on utf-8 characters? - https://phabricator.wikimedia.org/T401242 (10DamianZaremba) 03NEW [17:50:55] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [components-api,beta] CI pipelines should wait until Toolforge deployment is 100% successful - https://phabricator.wikimedia.org/T398485#11062120 (10dcaro) 05In progress→03Resolved @Sascha I'll close again, but do reopen if you see the issue... [17:55:47] 10Toolforge (Toolforge iteration 23): Logs that are present in `--follow` are missing when not `--follow` - https://phabricator.wikimedia.org/T401244 (10DamianZaremba) 03NEW [17:56:20] 10Toolforge (Toolforge iteration 23): Logs that are present in `--follow` are missing when not `--follow` - https://phabricator.wikimedia.org/T401244#11062159 (10DamianZaremba) [18:12:38] 06cloud-services-team, 10Toolforge: toolforge jobs logs throws an exception on utf-8 characters? - https://phabricator.wikimedia.org/T401242#11062212 (10DamianZaremba) Here is another instance: ` 2025-08-05T17:45:11+00:00 [bot-8d9fd6bdc-5zjd7] [job] [2025-08-05 17:45:11] cluebotng.INFO: Processing: [[July 22]]... [18:14:43] 06cloud-services-team, 10Toolforge: `toolforge jobs logs` appears to break on utf-8 characters - https://phabricator.wikimedia.org/T401242#11062232 (10DamianZaremba) [18:15:19] 06cloud-services-team, 10Toolforge: Logs that are present in `--follow` are missing when not `--follow` - https://phabricator.wikimedia.org/T401244#11062235 (10DamianZaremba) [18:32:07] 06cloud-services-team, 10Toolforge: Investigate daily disconnections of IRC bots hosted in Toolforge - https://phabricator.wikimedia.org/T400223#11062290 (10Danilo) Maybe it helps to find what is causing that by comparing nodes where it happens with nodes where it not happens. I know two bots that I do not see... [19:09:13] (03PS2) 10Gopavasanth: Enhanced layout structure with flexbox for better footer positioning [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1175859 (https://phabricator.wikimedia.org/T365438) [19:09:45] (03CR) 10Gopavasanth: "Resolved" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1175859 (https://phabricator.wikimedia.org/T365438) (owner: 10Gopavasanth) [19:10:09] (03CR) 10Gopavasanth: "resolved" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1175859 (https://phabricator.wikimedia.org/T365438) (owner: 10Gopavasanth) [19:45:11] (03CR) 10Eugene233: [C:03+2] Enhanced layout structure with flexbox for better footer positioning [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1175859 (https://phabricator.wikimedia.org/T365438) (owner: 10Gopavasanth) [19:45:36] (03Merged) 10jenkins-bot: Enhanced layout structure with flexbox for better footer positioning [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1175859 (https://phabricator.wikimedia.org/T365438) (owner: 10Gopavasanth) [20:25:55] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [20:26:28] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:30:48] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2010-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [20:30:56] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [20:31:28] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:41:39] (03PS1) 10BCornwall: Update Traffic repo locations to GitLab [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175948 [20:42:46] (03PS2) 10BCornwall: Update Traffic repo locations to GitLab [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175948 (https://phabricator.wikimedia.org/T347623) [21:00:48] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [21:01:22] (03PS1) 10BCornwall: Add more Traffic repositories [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175950 [21:10:48] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [21:38:56] 10Tool-masto-collab: Masto-Collab: Failing with 500 error when post contains image - https://phabricator.wikimedia.org/T401259 (10Peachey88) 03NEW [22:43:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-elastic-5 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:46:55] (03close) 10alien4444: Add new i18n file - Kannada(kn) language [toolforge-repos/wstranclude] - 10https://gitlab.wikimedia.org/toolforge-repos/wstranclude/-/merge_requests/1 (owner: 10anzx) [22:47:04] (03reopen) 10alien4444: Add new i18n file - Kannada(kn) language [toolforge-repos/wstranclude] - 10https://gitlab.wikimedia.org/toolforge-repos/wstranclude/-/merge_requests/1 (owner: 10anzx) [22:53:32] (03approved) 10alien4444: Add new i18n file - Kannada(kn) language [toolforge-repos/wstranclude] - 10https://gitlab.wikimedia.org/toolforge-repos/wstranclude/-/merge_requests/1 (owner: 10anzx) [22:54:19] (03merge) 10alien4444: Add new i18n file - Kannada(kn) language [toolforge-repos/wstranclude] - 10https://gitlab.wikimedia.org/toolforge-repos/wstranclude/-/merge_requests/1 (owner: 10anzx) [22:58:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance tools-elastic-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources