[00:05:55] FIRING: MaxConntrack: Max conntrack at 83.97% on cloudvirt1067:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:55:56] RESOLVED: MaxConntrack: Max conntrack at 83.3% on cloudvirt1067:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [02:59:44] 06cloud-services-team, 10Cloud-VPS: trove: Unable to create user with IPv6 address as host - https://phabricator.wikimedia.org/T393760#10961775 (10Andrew) Opened upstream issue https://storyboard.openstack.org/#!/story/2011486 [06:33:58] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/48 [06:34:02] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/48 [06:34:11] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/105 [06:34:12] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/109 [06:34:21] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/109 [06:34:22] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/105 [06:34:29] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/72 [06:34:30] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/49 [06:34:37] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/49 [06:34:52] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/25 [06:34:53] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/72 [06:35:00] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/21 [06:35:13] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/106 [06:35:13] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/25 [06:35:19] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/21 [06:35:21] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/106 [06:35:24] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/25 [06:35:26] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/25 [06:35:37] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/111 [06:35:43] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/75 [06:35:56] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/111 [06:35:57] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/112 [06:35:59] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/26 [06:36:01] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/26 [06:36:15] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/176 [06:36:19] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/75 [06:36:23] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/177 [06:36:25] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/83 [06:36:25] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/112 [06:36:25] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/84 [06:36:41] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/23 [06:36:41] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/176 [06:36:42] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/177 [06:36:49] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/31 [06:36:50] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/70 [06:36:53] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/44 [06:36:59] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/58 [06:36:59] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/84 [06:37:00] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/83 [06:37:05] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/23 [06:37:05] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/137 [06:37:08] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/71 [06:37:12] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/31 [06:37:16] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/72 [06:37:21] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/70 [06:37:25] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/250 [06:37:29] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/45 [06:37:33] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/49 [06:37:37] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/58 [06:37:41] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/44 [06:37:45] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/137 [06:37:49] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/71 [06:37:53] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/72 [06:37:57] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/250 [06:38:01] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/49 [06:38:05] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/45 [07:01:30] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on gitlab-runners-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [07:43:20] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Instance accounts-appserver7 in account-creation-assistance cannot connect to internal NTP - https://phabricator.wikimedia.org/T398099#10962019 (10taavi) 05Open→03Resolved [08:13:32] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,infra] Centralized logging for Toolforge infrastructure logs - https://phabricator.wikimedia.org/T97861#10962070 (10dcaro) I'm not convinced that we are in such dire situation for the extra complexity to be worth maintaining, can we discuss i... [08:15:03] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,infra] Centralized logging for Toolforge infrastructure logs - https://phabricator.wikimedia.org/T97861#10962072 (10dcaro) {T383081} is a temporary hack to persist to some extent the logs of some of the critical components we have, it's not m... [08:18:56] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 21), 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, and 2 others: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features - https://phabricator.wikimedia.org/T393564#10962078 (10dcaro) 05In progres... [08:25:16] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [Epic,builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build... - https://phabricator.wikimedia.org/T194332#10962087 [08:33:51] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build pack... - https://phabricator.wikimedia.org/T194332#10962138 [08:39:21] 10VPS-project-Phabricator, 06collaboration-services: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10962146 (10A_smart_kitten) >>! In T397626#10958631, @Aklapper wrote: > May need a `./bin/phd restart` If that's what's responsi... [08:43:12] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/45 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:43:14] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/45 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:43:23] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/44 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:43:39] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/44 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:43:59] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/250 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:44:01] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/250 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:00] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/111 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:03] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/111 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:09] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/44 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:10] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/112 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:20] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/112 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:44] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/83 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:46] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/83 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:59] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/84 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:46:00] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/84 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:46:38] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/48 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:46:40] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/48 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:46:42] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/112 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:46:44] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/109 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:46:50] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/109 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:47:05] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/49 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:47:10] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/49 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:47:44] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/84 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:47:45] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/49 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:47:51] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/49 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:48:15] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/72 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:48:37] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/49 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:48:40] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/71 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:48:46] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/71 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:49:02] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/70 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:49:27] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/72 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:51:16] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.20-20250701084823-2e9d659e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/843 [08:51:19] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.20-20250701084823-2e9d659e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/843 [08:51:24] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.72-20250701084902-2d07181b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/844 [08:51:26] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.72-20250701084902-2d07181b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/844 [08:51:48] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.132-20250701084953-3691f51e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/845 [08:55:41] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [08:56:44] 06cloud-services-team, 10Toolforge: toolforge jobs API, schedule vs schedule_actual, and API behaviour change between Feb and June 2025 - https://phabricator.wikimedia.org/T398281 (10Addshore) 03NEW [08:57:47] 06cloud-services-team, 10Toolforge: toolforge jobs API, schedule vs schedule_actual, and API behaviour change between Feb and June 2025 - https://phabricator.wikimedia.org/T398281#10962183 (10Addshore) [08:59:45] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [09:07:44] (03update) 10taavi: logging: loki: Add network policy rule for object storage access [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/827 (https://phabricator.wikimedia.org/T386480) [09:07:44] (03update) 10taavi: logging: loki: Add second Loki instance for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/834 (https://phabricator.wikimedia.org/T386480 https://phabricator.wikimedia.org/T97861) [09:07:45] (03update) 10taavi: logging: alloy: Add routing for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/835 (https://phabricator.wikimedia.org/T386480 https://phabricator.wikimedia.org/T97861) [09:07:45] (03update) 10taavi: logging: alloy: Allow running on the entire cluster [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/836 (https://phabricator.wikimedia.org/T97861) [09:07:54] (03update) 10taavi: logging: alloy: Add routing for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/835 (https://phabricator.wikimedia.org/T386480 https://phabricator.wikimedia.org/T97861) [09:07:57] (03update) 10taavi: logging: alloy: Allow running on the entire cluster [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/836 (https://phabricator.wikimedia.org/T97861) [09:07:59] (03update) 10taavi: logging: loki: Add network policy rule for object storage access [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/827 (https://phabricator.wikimedia.org/T386480) [09:08:00] (03update) 10taavi: logging: loki: Add second Loki instance for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/834 (https://phabricator.wikimedia.org/T386480 https://phabricator.wikimedia.org/T97861) [09:08:31] FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-6 is lagging behind the primary, the current lag is 19h 53m 24s - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [09:09:34] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/137 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:09:37] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/137 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:10:15] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/75 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:10:19] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/75 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:10:45] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/58 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:10:48] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/58 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:11:34] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/31 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:11:41] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/31 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:11:59] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/23 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:12:06] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/23 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:12:47] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/176 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:12:50] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/176 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:13:20] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/25 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:13:22] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/25 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:13:41] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/21 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:13:43] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/21 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:15:54] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/105 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:15:58] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/105 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:16:45] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/25 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:16:45] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/25 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:17:24] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.61-20250701091331-75b358e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/846 [09:17:37] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.68-20250701091200-4da32ba0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/847 [09:19:19] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.383-20250701091439-21cf74d2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/848 [09:19:22] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.383-20250701091439-21cf74d2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/848 [09:19:31] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.61-20250701091457-55540e65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/849 [09:19:34] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.61-20250701091457-55540e65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/849 [09:21:24] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.129-20250701091833-d59b8807 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/850 [09:22:58] RESOLVED: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-6 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:25:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [09:25:11] dcaro@cloudcumin1001: Unknown project "tools" [09:25:12] dcaro@cloudcumin1001: Did you mean to say "tools.tools" instead? [09:25:13] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.71-20250701092101-59b7337d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/851 [09:29:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [09:31:08] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.62-20250701092808-3c3a9b27 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/852 [09:39:47] (03approved) 10dcaro: builds-builder: bump to 0.0.132-20250701084953-3691f51e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/845 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:39:49] (03merge) 10dcaro: builds-builder: bump to 0.0.132-20250701084953-3691f51e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/845 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:39:59] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [09:47:13] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [09:47:38] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [09:56:08] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [09:56:45] (03approved) 10dcaro: registry-admission: bump to 0.0.62-20250701092808-3c3a9b27 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/852 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:56:49] (03update) 10dcaro: registry-admission: bump to 0.0.62-20250701092808-3c3a9b27 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/852 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:57:07] (03merge) 10dcaro: registry-admission: bump to 0.0.62-20250701092808-3c3a9b27 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/852 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:57:17] (03update) 10dcaro: envvars-api: bump to 0.0.71-20250701092101-59b7337d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/851 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:57:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [09:58:28] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [10:02:22] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [10:03:39] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [10:04:48] (03approved) 10dcaro: envvars-api: bump to 0.0.71-20250701092101-59b7337d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/851 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:04:51] (03merge) 10dcaro: envvars-api: bump to 0.0.71-20250701092101-59b7337d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/851 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:05:16] (03update) 10dcaro: components-api: bump to 0.0.129-20250701091833-d59b8807 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/850 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:05:21] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [10:05:35] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.30-20250701091216-052daa6d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/853 [10:06:13] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision request - Toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285 (10taavi) 03NEW [10:06:38] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,infra] Centralized logging for Toolforge infrastructure logs - https://phabricator.wikimedia.org/T97861#10962392 (10taavi) Sure, I created {T398285}. [10:08:49] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [10:09:42] (03update) 10dcaro: Refresh vcr tests and fix recording script [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/67 [10:11:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [10:13:36] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.195-20250701100648-3439fee6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/854 [10:15:38] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [10:23:03] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision request - Toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10962459 (10dcaro) > [1]: for example the rsyslog solution mentioned in T97861#10957908 still requires access to the Kubernetes API to function prope... [10:23:30] (03approved) 10dcaro: components-api: bump to 0.0.129-20250701091833-d59b8807 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/850 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:23:33] (03merge) 10dcaro: components-api: bump to 0.0.129-20250701091833-d59b8807 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/850 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:23:37] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,infra] Centralized logging for Toolforge infrastructure logs - https://phabricator.wikimedia.org/T97861#10962469 (10dcaro) Thanks! [10:24:40] (03approved) 10dcaro: Refresh vcr tests and fix recording script [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/67 [10:24:44] (03merge) 10dcaro: Refresh vcr tests and fix recording script [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/67 [10:25:52] (03update) 10dcaro: builds-api: bump to 0.0.195-20250701100648-3439fee6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/854 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:25:54] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [10:30:15] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [10:35:22] (03open) 10fnegri: dns: point tools-readonly.db.svc to new replica [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/252 (https://phabricator.wikimedia.org/T398170) [10:37:12] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.176-20250701103454-5ce4554a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/855 [11:09:19] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision request - Toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10962610 (10taavi) >>! In T398285#10962459, @dcaro wrote: >> [1]: for example the rsyslog solution mentioned in T97861#10957908 still requires access... [11:36:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [11:38:24] (03update) 10taavi: logging: loki: Add network policy rule for object storage access [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/827 (https://phabricator.wikimedia.org/T386480) [11:38:31] (03update) 10taavi: logging: loki: Add network policy rule for object storage access [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/827 (https://phabricator.wikimedia.org/T386480) [11:40:15] (03update) 10taavi: logging: Deploy tools Loki buckets [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/55 (https://phabricator.wikimedia.org/T386480 https://phabricator.wikimedia.org/T97861) [11:40:22] (03update) 10taavi: logging: Deploy tools Loki buckets [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/55 (https://phabricator.wikimedia.org/T97861) [11:41:00] (03update) 10taavi: logging: Deploy tools Loki buckets [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/55 (https://phabricator.wikimedia.org/T97861) [11:41:20] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [11:42:04] (03merge) 10taavi: logging: loki: Add network policy rule for object storage access [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/827 (https://phabricator.wikimedia.org/T386480) [11:42:08] (03update) 10taavi: logging: loki: Add second Loki instance for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/834 (https://phabricator.wikimedia.org/T386480 https://phabricator.wikimedia.org/T97861) [11:43:37] (03update) 10taavi: logging: loki: Add second Loki instance for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/834 (https://phabricator.wikimedia.org/T386480 https://phabricator.wikimedia.org/T97861) [11:43:38] (03update) 10taavi: logging: alloy: Add routing for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/835 (https://phabricator.wikimedia.org/T386480 https://phabricator.wikimedia.org/T97861) [11:43:41] (03update) 10taavi: logging: alloy: Allow running on the entire cluster [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/836 (https://phabricator.wikimedia.org/T386480 https://phabricator.wikimedia.org/T97861) [11:43:48] (03update) 10taavi: logging: alloy: Add routing for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/835 (https://phabricator.wikimedia.org/T386480) [11:43:48] (03update) 10taavi: logging: alloy: Allow running on the entire cluster [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/836 (https://phabricator.wikimedia.org/T386480) [11:43:52] (03update) 10taavi: logging: loki: Add second Loki instance for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/834 (https://phabricator.wikimedia.org/T386480) [11:44:17] (03merge) 10taavi: logging: Deploy tools Loki buckets [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/55 (https://phabricator.wikimedia.org/T97861) [11:44:44] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision request - Toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10962733 (10dcaro) > The documentation I linked says that the rsyslog Kubernetes metadata module seems to require connection details to the Kubernete... [11:45:32] (03approved) 10dcaro: builds-api: bump to 0.0.195-20250701100648-3439fee6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/854 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:45:35] (03update) 10dcaro: builds-api: bump to 0.0.195-20250701100648-3439fee6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/854 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:45:53] (03merge) 10dcaro: builds-api: bump to 0.0.195-20250701100648-3439fee6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/854 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:46:11] (03update) 10dcaro: envvars-admission: bump to 0.0.30-20250701091216-052daa6d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/853 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:46:13] (03approved) 10dcaro: envvars-admission: bump to 0.0.30-20250701091216-052daa6d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/853 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:46:16] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [11:51:19] 10Toolforge (Toolforge iteration 21): [lima-kilo] support range syntax for ansible tags - https://phabricator.wikimedia.org/T398306 (10Raymond_Ndibe) 03NEW [11:52:05] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/72 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:52:22] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [11:53:44] 06cloud-services-team, 10Striker, 10Continuous-Integration-Infrastructure, 10Toolhub, and 2 others: Multiple *-pipeline-test jobs failing to load pipelinelib with git error - https://phabricator.wikimedia.org/T386755#10962815 (10Nikerabbit) [11:55:00] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/72 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:55:02] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/72 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:55:28] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [11:56:20] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/70 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:56:22] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/70 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:56:35] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/177 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:57:33] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.177-20250701115518-24eb32ea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/855 [11:57:37] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.177-20250701115518-24eb32ea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/855 [11:58:46] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.73-20250701115634-84ac0cde [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/844 [11:58:49] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.73-20250701115634-84ac0cde [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/844 [12:03:22] (03open) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [12:03:28] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [12:03:37] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [12:08:24] (03open) 10taavi: logging: Deploy to tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856 (https://phabricator.wikimedia.org/T386480) [12:08:26] (03update) 10taavi: logging: Deploy to tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856 (https://phabricator.wikimedia.org/T386480) [12:09:59] (03update) 10taavi: logging: Deploy to tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856 (https://phabricator.wikimedia.org/T386480) [12:14:48] 10Cloud-VPS (Project-requests): Request creation of torrents VPS project - https://phabricator.wikimedia.org/T397861#10962913 (10TheresNoTime) Thanks for the comments @Andrew - `dumpstorrents` sounds like a better name, agreed - the [[ https://phabricator.wikimedia.org/T29653#4486036 | concerns raised here ]]... [12:16:34] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [o11y,logging,infra] Deploy Loki to store Toolforge tool log data - https://phabricator.wikimedia.org/T386480#10962929 (10taavi) [12:28:24] 10Cloud-VPS (Project-requests): Request creation of dumpstorrents VPS project - https://phabricator.wikimedia.org/T397861#10963009 (10Aklapper) [12:29:33] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision request - Toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10963016 (10taavi) [12:31:51] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision request - Toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10963027 (10taavi) >>! In T398285#10962733, @dcaro wrote: >> The documentation I linked says that the rsyslog Kubernetes metadata module seems to req... [12:45:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [12:51:26] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [12:53:51] (03merge) 10dcaro: envvars-admission: bump to 0.0.30-20250701091216-052daa6d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/853 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:54:07] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [12:57:06] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [13:12:08] (03merge) 10dcaro: docker: use the .svc.toolforge.org registry name [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/62 [13:15:57] (03open) 10dcaro: test: run pre-commit on ci [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/63 [13:15:57] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [13:17:21] (03update) 10dcaro: test: run pre-commit on ci [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/63 [13:18:56] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [13:26:05] (03update) 10dcaro: scheduled: add scheduled component support [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T395071) [13:26:28] (03approved) 10dcaro: api-gateway: bump to 0.0.73-20250701115634-84ac0cde [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/844 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:26:30] (03update) 10dcaro: api-gateway: bump to 0.0.73-20250701115634-84ac0cde [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/844 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:26:50] (03merge) 10dcaro: api-gateway: bump to 0.0.73-20250701115634-84ac0cde [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/844 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:27:02] (03update) 10dcaro: maintain-harbor: bump to 0.0.20-20250701084823-2e9d659e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/843 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:27:07] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [13:27:43] (03update) 10raymond-ndibe: scheduled: add scheduled component support [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T395071) (owner: 10dcaro) [13:27:47] (03approved) 10raymond-ndibe: scheduled: add scheduled component support [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T395071) (owner: 10dcaro) [13:29:34] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [13:33:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [13:35:20] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [13:48:34] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [13:51:48] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [13:52:14] (03approved) 10dcaro: maintain-harbor: bump to 0.0.20-20250701084823-2e9d659e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/843 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:52:16] (03merge) 10dcaro: maintain-harbor: bump to 0.0.20-20250701084823-2e9d659e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/843 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:52:55] (03update) 10dcaro: jobs-emailer: bump to 0.0.61-20250701091331-75b358e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/846 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:52:57] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [13:53:00] (03update) 10taavi: logging: Deploy to tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856 (https://phabricator.wikimedia.org/T386480) [13:53:04] (03update) 10taavi: logging: Deploy to tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856 (https://phabricator.wikimedia.org/T386480) [14:05:26] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [14:05:38] (03approved) 10fnegri: logging: Deploy to tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856 (https://phabricator.wikimedia.org/T386480) (owner: 10taavi) [14:05:39] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, and 2 others: decommission cloudcephosd200[12]-dev.codfw.wmnet - https://phabricator.wikimedia.org/T397968#10963632 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [14:06:25] 10Toolforge (Toolforge iteration 21): [lima-kilo] docker-compose not found - https://phabricator.wikimedia.org/T398322 (10Raymond_Ndibe) 03NEW [14:06:31] 10Toolforge (Toolforge iteration 21): [lima-kilo] docker-compose not found - https://phabricator.wikimedia.org/T398322#10963648 (10Raymond_Ndibe) a:03Raymond_Ndibe [14:08:01] (03approved) 10taavi: dns: point tools-readonly.db.svc to new replica [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/252 (https://phabricator.wikimedia.org/T398170) (owner: 10fnegri) [14:08:17] (03approved) 10dcaro: dns: point tools-readonly.db.svc to new replica [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/252 (https://phabricator.wikimedia.org/T398170) (owner: 10fnegri) [14:09:17] (03approved) 10dcaro: logging: Deploy to tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856 (https://phabricator.wikimedia.org/T386480) (owner: 10taavi) [14:09:31] (03merge) 10taavi: logging: Deploy to tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856 (https://phabricator.wikimedia.org/T386480) [14:10:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [14:10:59] (03open) 10taavi: logging: Add missing environment config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/857 [14:11:03] (03update) 10taavi: logging: Add missing environment config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/857 [14:11:30] (03merge) 10taavi: logging: Add missing environment config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/857 [14:11:37] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcephosd200[567]-dev - https://phabricator.wikimedia.org/T393614#10963659 (10Jhancock.wm) decom is completed. additional cable for cloudcephosd2007-dev is connected to port 23 [14:13:01] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [14:15:35] (03open) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] (ensure_use_of_amd64_arch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/252 (https://phabricator.wikimedia.org/T398322) [14:15:51] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision request - Reuse toolforge user tools central logging for toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10963718 (10dcaro) [14:16:07] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] (use_the_right_docker_compose) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [14:16:09] (03merge) 10fnegri: dns: point tools-readonly.db.svc to new replica [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/252 (https://phabricator.wikimedia.org/T398170) [14:16:27] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision request - Reuse toolforge user tools central logging for toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10963723 (10dcaro) [14:16:36] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:17:16] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:20:12] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision request - Reuse toolforge user tools central logging for toolforge infrastructure logging - https://phabricator.wikimedia.org/T398285#10963739 (10dcaro) Updated the task, reworded some stuff too to reflect the new option A differences a... [14:20:14] (03update) 10raymond-ndibe: docker-compose] use `docker compose` instead of docker-compose [repos/cloud/toolforge/lima-kilo] (ensure_use_of_amd64_arch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/252 (https://phabricator.wikimedia.org/T398322) [14:21:58] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 13Patch-For-Review: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2025-06-30 - https://phabricator.wikimedia.org/T398170#10963742 (10fnegri) tools-db-6 is the new replica host, replicating from the primary tools-db-4. I updated the DNS `tools-readonly... [14:26:10] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [14:26:25] (03approved) 10dcaro: jobs-emailer: bump to 0.0.61-20250701091331-75b358e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/846 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:26:30] (03update) 10dcaro: jobs-emailer: bump to 0.0.61-20250701091331-75b358e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/846 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:26:47] (03merge) 10dcaro: jobs-emailer: bump to 0.0.61-20250701091331-75b358e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/846 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:26:48] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [14:29:27] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [14:29:28] FIRING: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-5 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:29:44] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [14:30:07] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 (T398170) [14:30:11] T398170: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2025-06-30 - https://phabricator.wikimedia.org/T398170 [14:31:02] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 (T398170) [14:32:48] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [14:39:44] (03approved) 10dcaro: volume-admission: bump to 0.0.68-20250701091200-4da32ba0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/847 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:39:49] (03update) 10dcaro: volume-admission: bump to 0.0.68-20250701091200-4da32ba0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/847 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:40:02] (03merge) 10dcaro: volume-admission: bump to 0.0.68-20250701091200-4da32ba0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/847 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:40:43] (03update) 10dcaro: jobs-api: bump to 0.0.383-20250701091439-21cf74d2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/848 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:40:54] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:41:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:43:01] RESOLVED: ToolsToolsDBReplicationMissing: ToolsDB replication is not running on tools-db-5 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing [14:43:01] RESOLVED: ToolsToolsDBReplicationError: ToolsDB replication is broken on tools-db-5 (errno 1032) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [14:46:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:48:23] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:50:22] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:50:58] RESOLVED: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-5 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:53:54] (03open) 10dcaro: build: enable ci builds [repos/cloud/toolforge/toolforge-misctools] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-misctools/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) [14:55:18] (03update) 10raymond-ndibe: test: run pre-commit on ci [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/63 (owner: 10dcaro) [14:55:21] (03approved) 10raymond-ndibe: test: run pre-commit on ci [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/63 (owner: 10dcaro) [14:55:37] (03merge) 10dcaro: test: run pre-commit on ci [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/63 [14:58:30] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:09:23] (03open) 10taavi: logging: alloy: Deploy to two more tools workers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858 (https://phabricator.wikimedia.org/T386480) [15:09:24] (03update) 10taavi: logging: alloy: Deploy to two more tools workers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858 (https://phabricator.wikimedia.org/T386480) [15:13:24] (03update) 10taavi: logging: alloy: Deploy to two more tools workers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858 (https://phabricator.wikimedia.org/T386480) [15:13:54] (03update) 10dcaro: build: enable ci builds [repos/cloud/toolforge/toolforge-misctools] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-misctools/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) [15:14:07] (03approved) 10dcaro: logging: alloy: Deploy to two more tools workers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858 (https://phabricator.wikimedia.org/T386480) (owner: 10taavi) [15:14:35] (03approved) 10dcaro: jobs-api: bump to 0.0.383-20250701091439-21cf74d2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/848 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:14:37] (03merge) 10dcaro: jobs-api: bump to 0.0.383-20250701091439-21cf74d2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/848 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:15:03] (03update) 10dcaro: ingress-admission: bump to 0.0.61-20250701091457-55540e65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/849 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:15:05] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [15:15:08] (03update) 10taavi: logging: alloy: Deploy to two more tools workers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858 (https://phabricator.wikimedia.org/T386480) [15:15:38] (03merge) 10taavi: logging: alloy: Deploy to two more tools workers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858 (https://phabricator.wikimedia.org/T386480) [15:15:56] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logging [15:16:01] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging [15:16:13] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logging [15:21:04] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [15:21:46] (03update) 10dcaro: build: enable ci builds [repos/cloud/toolforge/toolforge-misctools] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-misctools/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) [15:22:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [15:23:10] (03update) 10dcaro: ingress-admission: bump to 0.0.61-20250701091457-55540e65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/849 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:23:15] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission [15:28:50] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10964303 (10ayounsi) @Andrew Would it be possible to use a single 25G uplink (cf. {T325531}) to make it better with automation and overall design (all reas... [15:31:31] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging [15:40:01] RESOLVED: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-6 is lagging behind the primary, the current lag is 1h 0m 26s - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [15:41:05] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [15:47:19] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [15:55:33] (03approved) 10dcaro: ingress-admission: bump to 0.0.61-20250701091457-55540e65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/849 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:40] (03merge) 10dcaro: ingress-admission: bump to 0.0.61-20250701091457-55540e65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/849 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:55:54] (03update) 10dcaro: maintain-kubeusers: bump to 0.0.177-20250701115518-24eb32ea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/855 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [15:57:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [16:05:51] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [16:07:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [16:10:02] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [16:14:00] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] (use_the_right_docker_compose) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [16:15:23] (03update) 10raymond-ndibe: [lima-kilo.toolforge] ensure use of amd64 arch [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/249 (https://phabricator.wikimedia.org/T398016) [16:17:08] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] (use_the_right_docker_compose) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [16:17:36] 10VPS-project-Phabricator, 06collaboration-services: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10964504 (10Dzahn) The `phd` service is masked on the instance (for some reason tbd). It can't be started unless it gets unmasked... [16:18:20] 10VPS-project-Phabricator, 06collaboration-services: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10964507 (10Dzahn) the reason for being masked is that it's masked.. which is of course very helpful. [16:19:29] (03update) 10raymond-ndibe: [start-devenv.sh] enable use of range syntax for ansible tags [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251 (https://phabricator.wikimedia.org/T398306) [16:19:48] (03update) 10raymond-ndibe: [lima-kilo.toolforge] ensure use of amd64 arch [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/249 (https://phabricator.wikimedia.org/T398016) [16:19:58] (03update) 10raymond-ndibe: docker-compose] use `docker compose` instead of docker-compose [repos/cloud/toolforge/lima-kilo] (ensure_use_of_amd64_arch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/252 (https://phabricator.wikimedia.org/T398322) [16:21:13] 10VPS-project-Phabricator, 06collaboration-services: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10964522 (10Dzahn) ` if $::fqdn == $active_server { ... $phd_service_ensure = 'running' ... } else { ...... [16:22:35] (03approved) 10raymond-ndibe: global: don't return tracebacks to users [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/102 (owner: 10dcaro) [16:22:41] (03update) 10raymond-ndibe: global: don't return tracebacks to users [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/102 (owner: 10dcaro) [16:23:21] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [16:23:41] 10VPS-project-Phabricator, 06collaboration-services: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10964530 (10Dzahn) But the $active_server is set in cloud VPS devtools project context: ` phabricator_active_server: phabricato... [16:24:19] 10VPS-project-Phabricator, 06collaboration-services: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10964532 (10Dzahn) a:03Dzahn [16:25:10] (03update) 10raymond-ndibe: cancel: add endpoint to cancel an ongoing deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/99 (https://phabricator.wikimedia.org/T395039) (owner: 10dcaro) [16:27:00] (03update) 10raymond-ndibe: tool-config: export the config schema [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/98 (https://phabricator.wikimedia.org/T397724) (owner: 10dcaro) [16:28:40] (03update) 10raymond-ndibe: components: add test for the generate feature [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/822 (owner: 10dcaro) [16:28:45] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [16:38:59] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [16:40:08] (03approved) 10dcaro: maintain-kubeusers: bump to 0.0.177-20250701115518-24eb32ea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/855 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:40:10] (03merge) 10dcaro: maintain-kubeusers: bump to 0.0.177-20250701115518-24eb32ea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/855 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:50:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2010-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:50:55] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcontrol2010-dev:9100 - https://phabricator.wikimedia.org/T398349 (10phaultfinder) 03NEW [16:56:53] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephosd2003-dev.codfw.wmnet - https://phabricator.wikimedia.org/T397979#10964716 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1003 for hosts: `cloudcephosd2003-dev.codfw.wmnet` - cloudceph... [16:59:02] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 13Patch-For-Review: decommission cloudcephosd2003-dev.codfw.wmnet - https://phabricator.wikimedia.org/T397979#10964717 (10Andrew) a:05Andrew→03None [16:59:40] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcephosd200[567]-dev - https://phabricator.wikimedia.org/T393614#10964721 (10Andrew) 05Open→03Resolved [17:05:48] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol2010-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:12:18] FIRING: [2x] PuppetFailure: Puppet has failed on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:12:22] FIRING: HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:12:24] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcontrol1006:9100 - https://phabricator.wikimedia.org/T398351 (10phaultfinder) 03NEW [17:15:28] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 06collaboration-services, 10GitLab (Infrastructure): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10964768 (10Dzahn) It kind of sounds from the last comment like this ticket still needs a foll... [17:17:18] FIRING: [4x] PuppetFailure: Puppet has failed on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:17:22] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:17:23] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T398353 (10phaultfinder) 03NEW [17:19:45] (03open) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:19:58] (03update) 10dcaro: build: enable ci builds [repos/cloud/toolforge/toolforge-misctools] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-misctools/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) [17:20:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 06collaboration-services, 10GitLab (Infrastructure): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10964816 (10Andrew) >>! In T396739#10964768, @Dzahn wrote: > It kind of sounds from the last c... [17:20:46] (03update) 10dcaro: build: enable ci builds [repos/cloud/toolforge/toolforge-misctools] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-misctools/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) [17:21:45] (03update) 10dcaro: build: enable ci builds [repos/cloud/toolforge/toolforge-misctools] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-misctools/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) [17:22:18] FIRING: [3x] PuppetFailure: Puppet has failed on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:26:49] (03update) 10raymond-ndibe: components: add test for the generate feature [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/822 (owner: 10dcaro) [17:27:18] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:30:04] (03update) 10dcaro: build: enable ci builds [repos/cloud/toolforge/toolforge-misctools] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-misctools/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) [17:30:46] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:31:27] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:31:43] (03update) 10dcaro: build: enable ci builds [repos/cloud/toolforge/toolforge-misctools] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-misctools/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) [17:33:27] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:34:22] 10Cloud-VPS (Project-requests): Request creation of dumpstorrents VPS project - https://phabricator.wikimedia.org/T397861#10964875 (10Andrew) >>! In T397861#10962913, @TheresNoTime wrote: > - the [[ https://phabricator.wikimedia.org/T29653#4486036 | concerns raised here ]] suggest that it //does// increase NFS... [17:34:52] (03update) 10ilanen1: Ilanmerge [toolforge-repos/miss-search] (update-cycle) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/8 [17:35:57] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:40:19] (03update) 10ilanen1: Ilanmerge [toolforge-repos/miss-search] (update-cycle) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/8 [17:41:27] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:42:15] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:43:32] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:45:28] (03update) 10raymond-ndibe: components: add test for the generate feature [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/822 (owner: 10dcaro) [17:45:29] (03approved) 10raymond-ndibe: components: add test for the generate feature [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/822 (owner: 10dcaro) [17:46:06] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:48:41] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 06collaboration-services, 10GitLab (Infrastructure): Volume is stuck to deleted instance in devtools project - https://phabricator.wikimedia.org/T396739#10964909 (10Dzahn) 05Resolved→03Open a:05Andrew→03Jelto @Jelto Should we deleted one of th... [17:52:33] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [17:57:58] (03update) 10dcaro: builder: allow specifying the arch [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/64 [18:10:13] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10964951 (10Andrew) >>! In T394333#10964303, @ayounsi wrote: > @Andrew Would it be possible to use a single 25G uplink (cf. {T325531}) to make it better wi... [18:22:00] (03update) 10raymond-ndibe: cancel: add new subcommand to cancel a deployment [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/45 (owner: 10dcaro) [18:22:18] (03update) 10raymond-ndibe: cancel: add new subcommand to cancel a deployment [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/45 (owner: 10dcaro) [18:22:22] (03approved) 10raymond-ndibe: cancel: add new subcommand to cancel a deployment [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/45 (owner: 10dcaro) [18:23:04] (03update) 10raymond-ndibe: cancel: add endpoint to cancel an ongoing deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/99 (https://phabricator.wikimedia.org/T395039) (owner: 10dcaro) [18:23:06] (03approved) 10raymond-ndibe: cancel: add endpoint to cancel an ongoing deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/99 (https://phabricator.wikimedia.org/T395039) (owner: 10dcaro) [18:26:20] (03update) 10raymond-ndibe: [api.jobs] health_check.type deprecation patch [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/173 (https://phabricator.wikimedia.org/T396210 https://phabricator.wikimedia.org/T396236) [18:30:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_osds [18:55:29] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.upgrade_osds (exit_code=0) [19:24:20] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: cloudcephosd200[567]-dev service implementation - https://phabricator.wikimedia.org/T397237#10965132 (10Andrew) 05Open→03Resolved [19:40:31] FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-6 is lagging behind the primary, the current lag is 1h 1m 25s - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [20:38:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.reboot_node [20:38:56] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.reboot_node (exit_code=99) [21:17:23] 10VPS-project-Phabricator, 06collaboration-services, 13Patch-For-Review: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10965445 (10Dzahn) @A_smart_kitten The `phd` service is now running on the instance. Did this fix it? [21:20:12] 10VPS-project-Phabricator, 06collaboration-services, 13Patch-For-Review: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10965446 (10A_smart_kitten) It looks like it did, ty! Interesting to know — it seems like the Phabricator... [21:27:01] 10VPS-project-Phabricator, 06collaboration-services, 13Patch-For-Review: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10965455 (10Dzahn) 05Open→03Resolved :) I am not sure if this has been broken forever since the tes... [21:34:23] 10VPS-project-Phabricator, 06collaboration-services: User "Recent Activity" feeds don't display anything on the Phabricator test instance - https://phabricator.wikimedia.org/T397626#10965488 (10A_smart_kitten) I think the instance('s database) was wiped/reset at some point (xref T265360#10598095). I'd gues...