[00:14:28] (03close) 10raymond-ndibe: [components-cli] no-op commit just to create a dummy release [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/14 [00:25:08] (03update) 10raymond-ndibe: [jobs-api] remove wait_for_job from runtime methods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/138 (https://phabricator.wikimedia.org/T359804) [00:35:57] (03open) 10raymond-ndibe: [jobs-cli] remove _delete_and_wait function [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/89 (https://phabricator.wikimedia.org/T359804) [00:36:56] (03update) 10raymond-ndibe: [jobs-cli] remove _delete_and_wait function [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/89 (https://phabricator.wikimedia.org/T359804) [00:37:06] (03update) 10raymond-ndibe: [jobs-cli] remove _delete_and_wait function [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/89 (https://phabricator.wikimedia.org/T359804) [00:39:12] 10Tool-schedule-deployment: IndexError: list index out of range on some windows - https://phabricator.wikimedia.org/T388633#10631290 (10bd808) Local testing shows that this is a different presentation of T388556, or maybe better stated I can reproduce this locally until I enable the fix for T388556 and then I ca... [00:39:28] 10Tool-schedule-deployment: Logic to find a deployment window given a unix timestamp assumes that wikitext uses the "SF" timezone marker - https://phabricator.wikimedia.org/T388556#10631292 (10bd808) [00:39:29] 10Tool-schedule-deployment: IndexError: list index out of range on some windows - https://phabricator.wikimedia.org/T388633#10631295 (10bd808) →14Duplicate dup:03T388556 [00:41:18] (03open) 10bd808: get_window_data: If SF window is not found, try CET [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/19 (https://phabricator.wikimedia.org/T388556) [00:42:24] (03approved) 10bd808: get_window_data: If SF window is not found, try CET [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/19 (https://phabricator.wikimedia.org/T388556) [00:42:30] (03merge) 10bd808: get_window_data: If SF window is not found, try CET [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/19 (https://phabricator.wikimedia.org/T388556) [00:43:57] 06cloud-services-team, 10Toolforge: [jobs-api,jobs-cli] API read timeout exception crashes `toolforge jobs logs --follow NAME` after a few seconds - https://phabricator.wikimedia.org/T358534#10631303 (10Raymond_Ndibe) This has been resolved [00:44:07] 06cloud-services-team, 10Toolforge: [jobs-api,jobs-cli] API read timeout exception crashes `toolforge jobs logs --follow NAME` after a few seconds - https://phabricator.wikimedia.org/T358534#10631304 (10Raymond_Ndibe) 05Open→03Resolved [00:45:09] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [toolforge,jobs] toolforge jobs logs read timeout error - https://phabricator.wikimedia.org/T356503#10631308 (10bd808) [00:45:10] 06cloud-services-team, 10Toolforge: [jobs-api,jobs-cli] API read timeout exception crashes `toolforge jobs logs --follow NAME` after a few seconds - https://phabricator.wikimedia.org/T358534#10631311 (10bd808) →14Duplicate dup:03T356503 [00:47:53] 10Tool-schedule-deployment, 13Patch-For-Review: Logic to find a deployment window given a unix timestamp assumes that wikitext uses the "SF" timezone marker - https://phabricator.wikimedia.org/T388556#10631315 (10bd808) 05In progress→03Resolved [02:44:55] 10VPS-project-Codesearch, 06collaboration-services, 13Patch-For-Review: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#10631458 (10Dzahn) [07:08:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-44 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:32:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:57:48] (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] (add_runner_parameter) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127) [09:29:55] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [10:12:10] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 [10:12:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:17:55] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 [10:17:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:48:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-44 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [10:48:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-44 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [10:51:51] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] Fix phrasing in "index starting up" error message [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1126585 (owner: 10Bartosz Dziewoński) [10:53:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-44 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [10:54:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-44 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [10:58:18] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-44 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:34:36] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:35:20] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [11:47:26] 10Horizon: Unable to login to horizon.w.o - error invalid request - https://phabricator.wikimedia.org/T388775 (10fgiunchedi) 03NEW [12:21:18] (03update) 10fnegri: Upgrade Kubernetes to 1.29 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/227 (https://phabricator.wikimedia.org/T362868) [12:21:23] (03update) 10fnegri: wmcs-k8s-metrics: upgrade charts for K8s v1.29 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/681 (https://phabricator.wikimedia.org/T362868) [12:22:51] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/10 [12:54:23] (03CR) 10Ladsgroup: [C:03+2] "Deploying frontend code is a bit complicated. I will try to do it a bit later today." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1126585 (owner: 10Bartosz Dziewoński) [12:55:33] (03Merged) 10jenkins-bot: Fix phrasing in "index starting up" error message [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1126585 (owner: 10Bartosz Dziewoński) [13:29:10] 06cloud-services-team, 10Toolforge: Restore Toolforge subfolder from backup - https://phabricator.wikimedia.org/T388713#10632707 (10MBH) 05Open→03Declined OK. [13:47:48] 10Tool-Gerrit-Patch-Uploader: Remove Bugzilla integration - https://phabricator.wikimedia.org/T86356#10632811 (10Aklapper) →14Duplicate dup:03T85701 [13:47:49] 10Tool-Gerrit-Patch-Uploader: Remove Bugzilla integration / notification - https://phabricator.wikimedia.org/T85701#10632813 (10Aklapper) [13:48:04] 10Horizon: Unable to login to horizon.w.o - error invalid request - https://phabricator.wikimedia.org/T388775#10632817 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Not sure exactly what changed, though I can login fine now! Resolving [13:51:30] (03approved) 10dcaro: [jobs-cli] remove _delete_and_wait function [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/89 (https://phabricator.wikimedia.org/T359804) (owner: 10raymond-ndibe) [13:51:33] (03update) 10dcaro: [jobs-cli] remove _delete_and_wait function [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/89 (https://phabricator.wikimedia.org/T359804) (owner: 10raymond-ndibe) [13:54:48] 10Tool-Gerrit-Patch-Uploader, 13Patch-For-Review: Update MediaWiki Logo on Gerrit Patch Uploader - https://phabricator.wikimedia.org/T297057#10632842 (10Aklapper) p:05Triage→03Low a:03Aklapper https://github.com/valhallasw/gerrit-patch-uploader/pull/41/ [13:55:52] (03approved) 10dcaro: deploy.sh: use helmfile sync instead of apply [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/712 (owner: 10fnegri) [13:56:43] (03approved) 10dcaro: Update to bookworm and new helmfile repo [repos/cloud/toolforge/packaging-helmfile] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/packaging-helmfile/-/merge_requests/2 (owner: 10fnegri) [14:00:31] (03update) 10dcaro: deploy_task: build the source based components [repos/cloud/toolforge/components-api] (add_deployment_status_updates) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/57 [14:01:34] (03merge) 10dcaro: deploy: support deploying source build components (buildpack) [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T384634) [14:01:35] (03update) 10dcaro: deployment: update the deployment state during deploy [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/54 [14:03:19] (03merge) 10fnegri: deploy.sh: use helmfile sync instead of apply [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/712 [14:04:29] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.83-20250313140145-6eef21b6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/713 (https://phabricator.wikimedia.org/T384634) [14:05:16] (03update) 10fnegri: Update to bookworm and new helmfile repo [repos/cloud/toolforge/packaging-helmfile] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/packaging-helmfile/-/merge_requests/2 [14:05:28] (03merge) 10fnegri: Update to bookworm and new helmfile repo [repos/cloud/toolforge/packaging-helmfile] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/packaging-helmfile/-/merge_requests/2 [14:05:58] (03update) 10fnegri: Update to bookworm and new helmfile repo [repos/cloud/toolforge/packaging-helmfile] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/packaging-helmfile/-/merge_requests/2 [14:09:41] (03update) 10dcaro: deployment: update the deployment state during deploy [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/54 [14:16:53] (03update) 10dcaro: deployment: update the deployment state during deploy [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/54 [14:23:18] (03update) 10dcaro: deployment: update the deployment state during deploy [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/54 [14:26:05] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [14:26:11] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [14:40:52] (03open) 10fnegri: Ugly hack to fix error with --wait [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 [14:40:56] 06cloud-services-team, 10Toolforge: [harbor,infra] Find a way to manage toolforge project policies with code - https://phabricator.wikimedia.org/T360509#10633102 (10Raymond_Ndibe) a:03Raymond_Ndibe [14:41:05] (03update) 10fnegri: deploy.sh: Ugly hack to fix error with --wait [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 [14:44:22] 06cloud-services-team, 10Toolforge: toolforge jobs load flushes out all jobs - https://phabricator.wikimedia.org/T364204#10633115 (10Raymond_Ndibe) This has been resolved since January [14:44:46] 06cloud-services-team, 10Toolforge: toolforge jobs load flushes out all jobs - https://phabricator.wikimedia.org/T364204#10633118 (10Raymond_Ndibe) 05Open→03Resolved [14:46:13] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18): Some toolforge components are running an old version - https://phabricator.wikimedia.org/T388797 (10fnegri) 03NEW [14:46:26] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18): Some toolforge components are running an old version - https://phabricator.wikimedia.org/T388797#10633132 (10fnegri) 05Open→03In progress [14:46:55] 10Toolforge (Toolforge iteration 18): [toolforge,jobs] "toolforge jobs logs" fails when job has not started yet - https://phabricator.wikimedia.org/T349775#10633151 (10Raymond_Ndibe) [14:47:02] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 05Goal: [harbor] Move harbor data to object storage service - https://phabricator.wikimedia.org/T350687#10633153 (10Raymond_Ndibe) [14:47:31] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 07Epic: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#10633167 (10Raymond_Ndibe) [14:48:10] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18): Some toolforge components are running an old version - https://phabricator.wikimedia.org/T388797#10633170 (10fnegri) I replaced `helmfile apply` with `helmfile diff` + `helmfile sync` in https://gitlab.wikimedia.org/repos/cloud/to... [14:50:52] 06cloud-services-team, 10Toolforge: [builds-api] Improve error message when logs time out - https://phabricator.wikimedia.org/T354755#10633181 (10Raymond_Ndibe) Not sure this is still a problem so I'm closing it. We've worked on so many things to improve errors like this, from increasing the timeout time to st... [14:51:03] 06cloud-services-team, 10Toolforge: [builds-api] Improve error message when logs time out - https://phabricator.wikimedia.org/T354755#10633184 (10Raymond_Ndibe) 05Open→03Resolved [14:51:30] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 05Goal: [harbor] Move harbor data to object storage service - https://phabricator.wikimedia.org/T350687#10633203 (10Raymond_Ndibe) 05Open→03Stalled [14:51:38] 06cloud-services-team, 10Toolforge (Toolforge iteration 18), 07Epic: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#10633206 (10Raymond_Ndibe) 05Open→03In progress [14:59:11] (03update) 10fnegri: deploy.sh: Ugly hack to fix error with --wait [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 [15:34:05] (03update) 10fnegri: deploy.sh: Ugly hack to fix error with --wait [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 [15:46:38] (03update) 10dcaro: deploy.sh: Ugly hack to fix error with --wait [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 (owner: 10fnegri) [15:46:51] (03approved) 10dcaro: deploy.sh: Ugly hack to fix error with --wait [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 (owner: 10fnegri) [15:48:11] (03update) 10dcaro: deploy.sh: Ugly hack to fix error with --wait [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 (owner: 10fnegri) [15:52:16] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [15:55:08] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [15:55:29] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [15:57:09] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [15:58:01] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [15:59:14] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:00:15] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:01:05] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18): Some toolforge components are running an old version - https://phabricator.wikimedia.org/T388797#10633590 (10dcaro) The fix for the above was to (for future reference): * Get a backup of the validatingwebhookconfiguration and mut... [16:01:47] (03merge) 10fnegri: deploy.sh: Ugly hack to fix error with --wait [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/714 [16:02:09] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:02:48] (03update) 10dcaro: components-api: bump to 0.0.83-20250313140145-6eef21b6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/713 (https://phabricator.wikimedia.org/T384634) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:02:56] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [16:04:06] 10Toolforge (Quota-requests): Request increased quota for milhistbot toolforge tool - https://phabricator.wikimedia.org/T387950#10633599 (10dcaro) a:03dcaro [16:04:22] 10Toolforge (Quota-requests): Request increased quota for milhistbot toolforge tool - https://phabricator.wikimedia.org/T387950#10633603 (10dcaro) [16:12:01] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:12:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [16:12:26] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:12:41] (03approved) 10dcaro: components-api: bump to 0.0.83-20250313140145-6eef21b6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/713 (https://phabricator.wikimedia.org/T384634) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:12:45] (03merge) 10dcaro: components-api: bump to 0.0.83-20250313140145-6eef21b6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/713 (https://phabricator.wikimedia.org/T384634) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:12:57] (03merge) 10dcaro: deployment: update the deployment state during deploy [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/54 [16:12:59] (03update) 10dcaro: deploy_task: build the source based components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/57 [16:14:01] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T362868) [16:14:04] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [16:14:40] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:15:07] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.84-20250313161312-ec70b457 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/715 (https://phabricator.wikimedia.org/T384634) [16:15:12] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.84-20250313161312-ec70b457 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/715 (https://phabricator.wikimedia.org/T384634) [16:16:44] (03update) 10dcaro: deploy_task: build the source based components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/57 [16:17:41] (03update) 10fnegri: volume-admission: bump to 0.0.64-20250311141825-ae6fee03 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/706 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:18:11] (03update) 10fnegri: ingress-admission: bump to 0.0.58-20250311141819-fc4d75c0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/707 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:18:15] (03update) 10fnegri: envvars-api: bump to 0.0.66-20250311141810-1e6c250b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/708 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:18:18] (03update) 10fnegri: registry-admission: bump to 0.0.58-20250311141826-b329eb93 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/709 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:18:19] (03update) 10fnegri: envvars-admission: bump to 0.0.26-20250311143252-34db248f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/710 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:18:31] (03update) 10fnegri: builds-api: bump to 0.0.181-20250311143252-180a03bb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/711 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:20:58] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [16:24:15] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:24:59] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T362868) [16:25:02] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [16:25:34] (03update) 10dcaro: deploy_task: build the source based components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/57 [16:30:46] (03update) 10fnegri: volume-admission: bump to 0.0.64-20250311141825-ae6fee03 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/706 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:30:54] (03merge) 10fnegri: volume-admission: bump to 0.0.64-20250311141825-ae6fee03 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/706 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:31:11] (03update) 10fnegri: ingress-admission: bump to 0.0.58-20250311141819-fc4d75c0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/707 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:31:24] (03merge) 10dcaro: deployment: add the status and long_status fields [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/20 [16:31:26] (03update) 10dcaro: deployment.list: tabulate and add some colors [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/21 [16:31:28] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [16:31:33] (03merge) 10dcaro: deployment.list: tabulate and add some colors [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/21 [16:31:35] (03update) 10dcaro: add build status [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/22 [16:32:17] (03approved) 10dcaro: components-api: bump to 0.0.84-20250313161312-ec70b457 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/715 (https://phabricator.wikimedia.org/T384634) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:32:27] (03update) 10dcaro: components-api: bump to 0.0.84-20250313161312-ec70b457 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/715 (https://phabricator.wikimedia.org/T384634) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:32:48] (03merge) 10dcaro: components-api: bump to 0.0.84-20250313161312-ec70b457 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/715 (https://phabricator.wikimedia.org/T384634) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:33:42] (03merge) 10dcaro: deploy_task: build the source based components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/57 [16:33:45] (03update) 10dcaro: deployment: use just the timestamp + a bit of random for the id [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/58 [16:34:00] (03merge) 10dcaro: add build status [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/22 [16:34:02] (03update) 10dcaro: show: add some indentation for the long status [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/23 [16:34:33] (03merge) 10dcaro: show: add some indentation for the long status [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/23 [16:35:32] (03open) 10dcaro: d/changelog: bump to 0.0.5 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/24 (https://phabricator.wikimedia.org/T384634) [16:36:04] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.92-20250313163352-ed309aa5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/716 (https://phabricator.wikimedia.org/T384634) [16:36:07] 06cloud-services-team: ProbeDown - https://phabricator.wikimedia.org/T388379#10633818 (10cmooney) This should be ok now if you want to try re-enabling the check. ` cmooney@prometheus2005:~$ ping -c 2 -6 wan.cloudgw.codfw1dev.wikimediacloud.org. PING wan.cloudgw.codfw1dev.wikimediacloud.org.(wan.cloudgw.codfw1dev... [16:36:30] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission (T362868) [16:36:35] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [16:39:02] (03update) 10dcaro: deployment: use just the timestamp + a bit of random for the id [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/58 [16:40:49] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:40:57] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-cli [16:41:36] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:44:32] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli [16:45:58] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-cli [16:46:44] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission (T362868) [16:46:49] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [16:48:58] (03update) 10fnegri: ingress-admission: bump to 0.0.58-20250311141819-fc4d75c0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/707 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:49:16] (03update) 10fnegri: ingress-admission: bump to 0.0.58-20250311141819-fc4d75c0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/707 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:49:26] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli [16:49:58] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:51:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [16:53:46] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:55:47] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:56:12] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [16:56:36] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:57:30] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:58:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:02:03] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [17:04:28] (03merge) 10fnegri: ingress-admission: bump to 0.0.58-20250311141819-fc4d75c0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/707 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:04:52] (03update) 10fnegri: envvars-api: bump to 0.0.66-20250311141810-1e6c250b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/708 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:04:58] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api (T362868) [17:05:02] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [17:05:20] (03update) 10fnegri: envvars-api: bump to 0.0.66-20250311141810-1e6c250b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/708 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:07:03] (03open) 10dcaro: functional-tests: adapt components to use newer cli tables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/717 [17:09:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:09:25] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18): Some toolforge components are running an old version - https://phabricator.wikimedia.org/T388797#10633990 (10fnegri) 05In progress→03Resolved [17:13:57] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [17:14:03] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api (T362868) [17:14:06] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [17:15:51] (03merge) 10fnegri: envvars-api: bump to 0.0.66-20250311141810-1e6c250b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/708 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:16:13] (03update) 10fnegri: registry-admission: bump to 0.0.58-20250311141826-b329eb93 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/709 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:17:18] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission (T362868) [17:17:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:18:37] (03open) 10raymond-ndibe: [components-api] update repo [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/60 [17:20:37] (03update) 10raymond-ndibe: [components-api] update repo [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/60 [17:21:43] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [17:21:45] (03update) 10raymond-ndibe: [components-api] update poetry.lock [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/60 [17:24:01] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:24:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [17:24:38] (03approved) 10dcaro: components-api: bump to 0.0.92-20250313163352-ed309aa5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/716 (https://phabricator.wikimedia.org/T384634) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:24:43] (03update) 10dcaro: components-api: bump to 0.0.92-20250313163352-ed309aa5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/716 (https://phabricator.wikimedia.org/T384634) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:25:06] (03merge) 10dcaro: components-api: bump to 0.0.92-20250313163352-ed309aa5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/716 (https://phabricator.wikimedia.org/T384634) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:27:45] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission (T362868) [17:27:49] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [17:29:09] (03update) 10fnegri: registry-admission: bump to 0.0.58-20250311141826-b329eb93 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/709 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:29:30] (03merge) 10fnegri: registry-admission: bump to 0.0.58-20250311141826-b329eb93 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/709 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:29:45] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission (T362868) [17:30:13] (03update) 10fnegri: builds-api: bump to 0.0.181-20250311143252-180a03bb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/711 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:31:15] (03update) 10dcaro: functional-tests: adapt components to use newer cli tables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/717 [17:32:32] (03approved) 10dcaro: functional-tests: adapt components to use newer cli tables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/717 [17:32:40] (03update) 10dcaro: functional-tests: adapt components to use newer cli tables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/717 [17:32:59] (03merge) 10dcaro: functional-tests: adapt components to use newer cli tables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/717 [17:33:16] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-cli [17:37:27] (03Abandoned) 10David Caro: Revert "Add PSP roles for buildpack images" [labs/tools/maintain-kubeusers] - 10https://gerrit.wikimedia.org/r/853538 (owner: 10David Caro) [17:40:25] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission (T362868) [17:40:30] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [17:42:17] (03PS5) 10David Caro: wmcs.ceph.osd.show_info: adapt for long output [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/904770 [17:42:21] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli [17:46:44] (03approved) 10dcaro: d/changelog: bump to 0.0.5 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/24 (https://phabricator.wikimedia.org/T384634) [17:46:48] (03merge) 10dcaro: d/changelog: bump to 0.0.5 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/24 (https://phabricator.wikimedia.org/T384634) [17:48:51] (03update) 10fnegri: envvars-admission: bump to 0.0.26-20250311143252-34db248f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/710 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:49:41] (03merge) 10fnegri: envvars-admission: bump to 0.0.26-20250311143252-34db248f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/710 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:50:01] (03update) 10fnegri: builds-api: bump to 0.0.181-20250311143252-180a03bb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/711 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:50:09] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api (T362868) [17:50:14] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [17:53:06] (03open) 10bd808: home: Display local time for upcoming backport windows [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/20 (https://phabricator.wikimedia.org/T382417) [17:53:54] 10Tool-schedule-deployment, 13Patch-For-Review: Show upcoming backport windows in local time - https://phabricator.wikimedia.org/T382417#10634173 (10bd808) 05Open→03In progress a:03bd808 [17:55:21] (03approved) 10bd808: home: Display local time for upcoming backport windows [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/20 (https://phabricator.wikimedia.org/T382417) [17:55:24] (03merge) 10bd808: home: Display local time for upcoming backport windows [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/20 (https://phabricator.wikimedia.org/T382417) [17:58:32] 10Tool-schedule-deployment: ScheduleDeploymentBot should require confirmation to add more than 6 patches to a backport window - https://phabricator.wikimedia.org/T367229#10634208 (10bd808) >>! In T367229#10634202, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-cloud), href=https://sal.toolf... [18:00:36] 10Tool-schedule-deployment, 13Patch-For-Review: Show upcoming backport windows in local time - https://phabricator.wikimedia.org/T382417#10634218 (10bd808) 05In progress→03Resolved [18:00:56] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api (T362868) [18:01:00] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [18:04:09] (03merge) 10fnegri: builds-api: bump to 0.0.181-20250311143252-180a03bb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/711 (https://phabricator.wikimedia.org/T362868) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [18:04:57] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T362868) [18:05:37] (03update) 10fnegri: wmcs-k8s-metrics: upgrade charts for K8s v1.29 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/681 (https://phabricator.wikimedia.org/T362868) [18:14:29] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T362868) [18:14:38] T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868 [18:22:30] 10Toolforge (Toolforge iteration 18): [builds-api] Limit the amount of running builds - https://phabricator.wikimedia.org/T388706#10634291 (10dcaro) p:05Triage→03High [18:22:32] 10Toolforge (Toolforge iteration 18): [components-api] allow stopping a deployment that's running - https://phabricator.wikimedia.org/T388644#10634292 (10dcaro) p:05Triage→03High [18:22:38] 10Toolforge (Toolforge iteration 18): [components-api] restrict running deplpoyments to 1 - https://phabricator.wikimedia.org/T388643#10634293 (10dcaro) p:05Triage→03High [18:23:25] 10Toolforge (Toolforge iteration 18): [components-api] allow stopping a deployment that's running - https://phabricator.wikimedia.org/T388644#10634294 (10dcaro) [18:24:22] (03approved) 10dcaro: [components-api] update poetry.lock [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/60 (owner: 10raymond-ndibe) [18:25:48] (03merge) 10fnegri: wmcs-k8s-metrics: upgrade charts for K8s v1.29 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/681 (https://phabricator.wikimedia.org/T362868) [18:28:36] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868#10634300 (10fnegri) [18:28:55] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [18:29:05] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [18:35:10] 10Toolforge (Quota-requests): Request increased quota for milhistbot toolforge tool - https://phabricator.wikimedia.org/T387950#10634326 (10dcaro) p:05Triage→03Medium [18:40:42] 10Toolforge (Toolforge iteration 18): [components-api] use the component name for the image instead of the default tool - https://phabricator.wikimedia.org/T388830 (10dcaro) 03NEW [18:41:11] 10Toolforge (Toolforge iteration 18): [components-api] use the component name for the image instead of the default tool - https://phabricator.wikimedia.org/T388830#10634346 (10dcaro) p:05Triage→03High [18:41:23] 10Toolforge (Toolforge iteration 18): [components-api] use the component name for the image instead of the default tool - https://phabricator.wikimedia.org/T388830#10634348 (10dcaro) [18:41:27] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10634349 (10dcaro) [18:41:32] 10Toolforge (Toolforge iteration 18): [builds-api] Limit the amount of running builds - https://phabricator.wikimedia.org/T388706#10634350 (10dcaro) [18:41:35] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10634351 (10dcaro) [18:41:41] 10Toolforge (Toolforge iteration 18): [components-api] allow stopping a deployment that's running - https://phabricator.wikimedia.org/T388644#10634352 (10dcaro) [18:41:43] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10634353 (10dcaro) [18:41:59] 10Toolforge (Toolforge iteration 18): [components-api] restrict running deplpoyments to 1 - https://phabricator.wikimedia.org/T388643#10634354 (10dcaro) [18:42:01] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 18), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10634355 (10dcaro) [18:42:57] 10Toolforge (Toolforge iteration 18): [components-api] Trigger builds on every deploy - https://phabricator.wikimedia.org/T384634#10634357 (10dcaro) 05In progress→03Resolved [20:06:10] 10VPS-project-Codesearch: Codesearches are timing out (2025-03-11) - https://phabricator.wikimedia.org/T388545#10634609 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup somehow the paact log is now only 157MB which is quite nice. I call this done. [20:07:04] 10VPS-project-Codesearch: Codesearches are timing out (2025-03-11) - https://phabricator.wikimedia.org/T388545#10634612 (10A_smart_kitten) ty for fixing this, @ladsgroup :) [20:35:43] 10Tool-ldap, 07LDAP: "Service group" is incomplete - https://phabricator.wikimedia.org/T378049#10634647 (10bd808) The LDAP directory itself seems to be what is weird here. Inspecting the `uid=magnus` record's view of group memberships: `lang=shell-session,lines=10 $ ldap uid=magnus memberOf | grep memberOf: m... [20:36:29] 10Tool-ldap, 07LDAP: Not all expected `memberOf` group relationships are present for `uid=magnus` - https://phabricator.wikimedia.org/T378049#10634651 (10bd808) [21:01:15] (03CR) 10Ladsgroup: [C:03+2] "I think I deployed it." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1126585 (owner: 10Bartosz Dziewoński) [21:28:28] (03open) 10raymond-ndibe: [toolforge-deploy] increase milhistbot harbor quota to 2Gi [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/718 (https://phabricator.wikimedia.org/T387950) [22:16:37] (03approved) 10raymond-ndibe: [toolforge-deploy] increase milhistbot harbor quota to 2Gi [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/718 (https://phabricator.wikimedia.org/T387950) [22:16:41] (03merge) 10raymond-ndibe: [toolforge-deploy] increase milhistbot harbor quota to 2Gi [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/718 (https://phabricator.wikimedia.org/T387950) [22:17:29] (03open) 10raymond-ndibe: Revert "[toolforge-deploy] increase milhistbot harbor quota to 2Gi" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/719 [22:18:04] (03approved) 10raymond-ndibe: Revert "[toolforge-deploy] increase milhistbot harbor quota to 2Gi" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/719 [22:18:07] (03update) 10raymond-ndibe: Revert "[toolforge-deploy] increase milhistbot harbor quota to 2Gi" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/719 [22:18:20] (03merge) 10raymond-ndibe: Revert "[toolforge-deploy] increase milhistbot harbor quota to 2Gi" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/719 [22:19:48] (03open) 10raymond-ndibe: [toolforge-deploy] increase milhistbot harbor quota to 2Gi [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/720 (https://phabricator.wikimedia.org/T388274) [22:23:21] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [22:32:33] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [22:33:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [22:42:57] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [22:55:03] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/10 (owner: 10l10n-bot) [22:55:08] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/10 (owner: 10l10n-bot)