[00:11:02] <wikibugs>	 06Toolforge-standards-committee, 10video2commons: Write-access to Video2Commons GitHub repo - https://phabricator.wikimedia.org/T394802#11248228 (10bd808) 05Open→03Resolved a:03bd808 I wonder if @Don-vip actually tried to add you to the repo or not. I can see that he has the "role: admin" at https://...
[01:17:03] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[02:07:56] <jinxer-wm>	 FIRING: [2x] SystemdUnitDown: The service unit remove_dangling_cinder_snapshots.service is in failed status on host cloudbackup1001-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[02:58:32] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[04:02:56] <jinxer-wm>	 FIRING: [2x] SystemdUnitDown: The systemd unit remove_dangling_cinder_snapshots.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[04:03:05] <wikibugs>	 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406538 (10phaultfinder) 03NEW
[04:28:52] <wikibugs>	 (03PS1) 10Shivaansh Singh: Add one-click bug reporting system [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1193969 (https://phabricator.wikimedia.org/T288418)
[04:32:02] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1193833 (owner: 10L10n-bot)
[04:32:27] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1193834 (owner: 10L10n-bot)
[06:46:19] <wm-bot2>	 !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304356335 (https://github.com/cluebotng/component-configs/commits/49f902fb1380f69c5e0156466832b031a9c228ee)
[06:46:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL
[06:48:54] <wm-bot2>	 !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304366702 (https://github.com/cluebotng/component-configs/commits/5b83bca0e9293029698d7f3a1b2764727ae7f971)
[06:48:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL
[06:58:32] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[07:01:25] <wikibugs>	 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Create OAuth2 token in test Phabricator - https://phabricator.wikimedia.org/T406495#11248645 (10SLyngshede-WMF) @Aklapper that's fine, I just need a user to be able to sign in using Phabricator as an OAuth server, li...
[07:50:53] <wikibugs>	 (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:51:14] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:51:40] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/31 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:51:43] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/31 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:51:51] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/74 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:51:53] <wikibugs>	 (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/32 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:52:07] <wikibugs>	 (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/32 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:52:27] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/136 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:52:30] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/136 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:52:38] <wikibugs>	 (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/137 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:53:26] <wikibugs>	 (03approved) 10dcaro: openapi: add missing include_unset parameter [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/225
[07:53:28] <wikibugs>	 (03merge) 10dcaro: openapi: add missing include_unset parameter [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/225
[07:53:40] <wikibugs>	 (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/224 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:53:43] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/224 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:53:59] <wikibugs>	 (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/32 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:54:00] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.135-20251007075204-07d0a8b0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/985
[07:54:06] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.135-20251007075204-07d0a8b0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/985
[07:54:38] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.67-20251007075152-2ed321a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/986
[07:56:03] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.164-20251007075243-4a976e9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/987
[07:56:05] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.164-20251007075243-4a976e9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/987
[07:56:19] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/224 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:56:28] <wikibugs>	 (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/137 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:56:33] <wikibugs>	 (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/137 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:56:54] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.421-20251007075338-df9f4836 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/988
[07:56:54] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.421-20251007075338-df9f4836 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/988
[07:56:54] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.68-20251007075452-ba81d174 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/986
[07:57:05] <wikibugs>	 (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/223 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:57:06] <wikibugs>	 (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/223 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:57:07] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.68-20251007075452-ba81d174 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/986
[07:59:23] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.165-20251007075659-1c9e037d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/987
[07:59:27] <wikibugs>	 (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/223 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[07:59:32] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.165-20251007075659-1c9e037d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/987
[07:59:42] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
[07:59:42] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.422-20251007075639-ac7e0625 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/988
[07:59:45] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.422-20251007075639-ac7e0625 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/988
[08:02:34] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.423-20251007075945-b5b93756 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/988
[08:02:36] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.423-20251007075945-b5b93756 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/988
[08:06:12] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/146 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:06:16] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/146 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:06:48] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/30 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:06:53] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/30 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:06:58] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/77 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:01] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/77 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:02] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/26 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:08] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/26 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:12] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/26 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:17] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/26 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:28] <wikibugs>	 (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/79 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:32] <wikibugs>	 (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/79 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:41] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/37 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:44] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/37 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:48] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/65 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:07:53] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/65 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:08:00] <wikibugs>	 (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/80 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:08:10] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/80 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:08:36] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[08:08:36] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api
[08:08:46] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
[08:09:00] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer
[08:10:03] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/80 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:12:17] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.74-20251007080805-4d1a870d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/989
[08:12:19] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.74-20251007080805-4d1a870d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/989
[08:12:19] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.67-20251007080701-39739ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/990
[08:12:21] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.67-20251007080701-39739ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/990
[08:12:29] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
[08:12:50] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.33-20251007080733-bc76cfcc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/991
[08:12:54] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.33-20251007080733-bc76cfcc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/991
[08:13:00] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.66-20251007080726-c67edf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/992
[08:13:03] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.66-20251007080726-c67edf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/992
[08:13:15] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.80-20251007081037-0f1e1046 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/993
[08:13:19] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.80-20251007081037-0f1e1046 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/993
[08:13:44] <wikibugs>	 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Create OAuth2 token in test Phabricator - https://phabricator.wikimedia.org/T406495#11248781 (10Aklapper) @SLyngshede-WMF: Alright, for the start, please create an account on https://phabricator.wmcloud.org/ and then...
[08:14:59] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/53 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:15:04] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/53 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:17:31] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.60-20251007081516-cee655d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/994
[08:27:15] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
[08:36:03] <wm-bot2>	 !log godog@r5 toolsbeta START - Cookbook wmcs.nfs.migrate_service
[08:36:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[08:37:41] <wm-bot2>	 !log godog@r5 toolsbeta END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
[08:37:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[08:39:56] <wm-bot2>	 !log godog@r5 toolsbeta START - Cookbook wmcs.nfs.migrate_service
[08:39:56] <wm-bot2>	 !log godog@r5 toolsbeta END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
[08:39:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[08:40:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[08:40:41] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[08:42:12] <wikibugs>	 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering, 06Data-Platform-SRE: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11248853 (10Gehel) We're waiting on #data-engineering to validate.
[08:42:31] <wikibugs>	 (03update) 10dcaro: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) (owner: 10raymond-ndibe)
[08:42:37] <wikibugs>	 (03update) 10dcaro: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] (update_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) (owner: 10raymond-ndibe)
[08:43:27] <wmcs-alerts>	 FIRING: ToolsbetaNFSDown: No toolsbeta nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsbetaNFSDown
[08:44:39] <wikibugs>	 06Toolforge-standards-committee, 10video2commons: Write-access to Video2Commons GitHub repo - https://phabricator.wikimedia.org/T394802#11248878 (10Don-vip) Woops no, I didn't try as I genuinely thought I wasn't able to do it. Thank you Bryan!
[08:48:27] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
[08:48:42] <wmcs-alerts>	 RESOLVED: ToolsbetaNFSDown: No toolsbeta nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsbetaNFSDown
[08:50:41] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[08:51:19] <wikibugs>	 10Cloud-VPS (Project-requests), 10Wikidata, 10Wikidata-Platform, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Request creation of query-service (blazegraph alternatives) VPS project - https://phabricator.wikimedia.org/T406240#11248908 (10Gehel)
[08:52:47] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7
[08:53:26] <wikibugs>	 10Cloud Services Proposals, 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#11248934 (10Gehel)
[08:56:37] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7
[09:08:57] <wikibugs>	 (03open) 10fnegri: Update OpenStack provider [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/91
[09:08:58] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
[09:09:40] <wikibugs>	 (03PS1) 10Filippo Giunchedi: nfs: force volume state available during migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1194135 (https://phabricator.wikimedia.org/T404584)
[09:14:38] <wikibugs>	 (03approved) 10dcaro: jobs-emailer: bump to 0.0.68-20251007075452-ba81d174 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/986 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[09:14:41] <wikibugs>	 (03merge) 10dcaro: jobs-emailer: bump to 0.0.68-20251007075452-ba81d174 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/986 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[09:14:46] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[09:17:15] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
[09:43:06] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[09:47:00] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
[10:01:06] <wikibugs>	 (03approved) 10dcaro: maintain-harbor: bump to 0.0.60-20251007081516-cee655d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/994 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[10:01:12] <wikibugs>	 (03update) 10dcaro: maintain-harbor: bump to 0.0.60-20251007081516-cee655d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/994 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[10:01:25] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
[10:01:33] <wikibugs>	 (03merge) 10dcaro: maintain-harbor: bump to 0.0.60-20251007081516-cee655d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/994 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[10:05:41] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
[10:08:31] <wikibugs>	 (03PS2) 10Filippo Giunchedi: nfs: force volume state available during migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1194135 (https://phabricator.wikimedia.org/T404584)
[10:10:46] <wikibugs>	 (03CR) 10Filippo Giunchedi: "Slightly improved version to act only on actually 'reserved' volumes" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1194135 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi)
[10:11:27] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
[10:15:08] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
[10:17:45] <wikibugs>	 06cloud-services-team, 10Toolforge: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11249345 (10fnegri) I found some interesting things:  * I can reproduce the failure with `curl` from the GitLab worker VM (`runner-1033.gitlab-runners.eqiad1.wikimedia.cloud`), running...
[10:18:23] <wikibugs>	 10VPS-project-devtools, 06collaboration-services, 10GitLab, 13Patch-For-Review: Puppet failure on gitlab-1002.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T406234#11249346 (10Jelto) 05Open→03Resolved The puppet issue is fixed, I'll resolve the task.
[10:19:01] <wikibugs>	 06cloud-services-team, 10Toolforge: BlockedExternalDomain hits are not available in the toolforge replicas - https://phabricator.wikimedia.org/T406562 (10Count_Count) 03NEW
[10:19:23] <wikibugs>	 06cloud-services-team, 10Toolforge: BlockedExternalDomain hits are not available in the toolforge replicas - https://phabricator.wikimedia.org/T406562#11249360 (10Count_Count) a:03Ladsgroup
[10:20:24] <wikibugs>	 06cloud-services-team, 10Toolforge: BlockedExternalDomain hits are not available in the toolforge replicas - https://phabricator.wikimedia.org/T406562#11249362 (10Count_Count) Assigning to Ladsgroup as discussed here: https://meta.wikimedia.org/wiki/Talk:Spam_blacklist#c-Ladsgroup-20251006153200-Count_Count-20...
[10:20:34] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
[10:25:02] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
[10:31:40] <wikibugs>	 06cloud-services-team, 10Toolforge: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11249383 (10fnegri) I can also reproduce the error using `docker-registry.wikimedia.org/bookworm` or `debian:bookworm`.  I can NOT reproduce it using `docker-registry.wikimedia.org/trix...
[10:35:25] <wikibugs>	 (03open) 10fnegri: tofu-provisioning: use trixie base image [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/71 (https://phabricator.wikimedia.org/T405742)
[10:43:58] <wikibugs>	 06cloud-services-team, 10Data-Services: BlockedExternalDomain hits are not available in the toolforge replicas - https://phabricator.wikimedia.org/T406562#11249408 (10taavi)
[10:58:32] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[11:02:57] <wikibugs>	 (03update) 10fnegri: tofu-provisioning: use trixie base image [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/71 (https://phabricator.wikimedia.org/T405742)
[11:05:47] <wikibugs>	 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11249489 (10fnegri) 05Open→03In progress a:03fnegri
[11:19:37] <wikibugs>	 (03approved) 10dcaro: api-gateway: bump to 0.0.80-20251007081037-0f1e1046 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/993 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:19:42] <wikibugs>	 (03update) 10dcaro: api-gateway: bump to 0.0.80-20251007081037-0f1e1046 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/993 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:20:08] <wikibugs>	 (03merge) 10dcaro: api-gateway: bump to 0.0.80-20251007081037-0f1e1046 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/993 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:20:23] <wikibugs>	 (03update) 10dcaro: ingress-admission: bump to 0.0.66-20251007080726-c67edf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/992 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:20:32] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
[11:28:54] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
[11:31:01] <wikibugs>	 (03open) 10arian-ar: Add color scheme for dark mode support [toolforge-repos/wlm-iran] - 10https://gitlab.wikimedia.org/toolforge-repos/wlm-iran/-/merge_requests/1
[11:31:06] <wikibugs>	 (03merge) 10arian-ar: Add color scheme for dark mode support [toolforge-repos/wlm-iran] - 10https://gitlab.wikimedia.org/toolforge-repos/wlm-iran/-/merge_requests/1
[11:36:44] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
[11:44:51] <wm-bot2>	 !log godog@r5 toolsbeta START - Cookbook wmcs.nfs.migrate_service
[11:44:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[11:46:15] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
[11:46:32] <wm-bot2>	 !log godog@r5 toolsbeta END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99)
[11:46:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[11:49:41] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:52:35] <wm-bot2>	 !log godog@r5 toolsbeta START - Cookbook wmcs.nfs.migrate_service
[11:52:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[11:54:41] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:56:03] <wm-bot2>	 !log godog@r5 toolsbeta END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0)
[11:56:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[11:58:10] <wikibugs>	 (03PS3) 10Filippo Giunchedi: nfs: force volume state available during migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1194135 (https://phabricator.wikimedia.org/T404584)
[11:59:47] <wikibugs>	 (03CR) 10Filippo Giunchedi: "Last PS tested in toolsbeta, works as expected (i.e. cookbook runs unattended)" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1194135 (https://phabricator.wikimedia.org/T404584) (owner: 10Filippo Giunchedi)
[12:14:03] <wikibugs>	 (03approved) 10dcaro: ingress-admission: bump to 0.0.66-20251007080726-c67edf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/992 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:14:08] <wikibugs>	 (03merge) 10dcaro: ingress-admission: bump to 0.0.66-20251007080726-c67edf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/992 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:14:20] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
[12:22:11] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
[12:39:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-65 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[12:47:19] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
[12:47:37] <wikibugs>	 (03update) 10dcaro: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] (update_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) (owner: 10raymond-ndibe)
[12:51:22] <wikibugs>	 (03update) 10dcaro: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] (update_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) (owner: 10raymond-ndibe)
[12:56:12] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
[12:56:49] <wikibugs>	 (03approved) 10dcaro: envvars-admission: bump to 0.0.33-20251007080733-bc76cfcc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/991 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:56:53] <wikibugs>	 (03update) 10dcaro: envvars-admission: bump to 0.0.33-20251007080733-bc76cfcc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/991 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:57:09] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
[12:57:12] <wikibugs>	 (03merge) 10dcaro: envvars-admission: bump to 0.0.33-20251007080733-bc76cfcc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/991 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:58:17] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39
[12:59:22] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39
[13:03:31] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
[13:05:08] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
[13:13:13] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
[13:27:41] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:30:27] <wmcs-alerts>	 FIRING: ToolsbetaNFSDown: No toolsbeta nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsbetaNFSDown
[13:34:58] <wikibugs>	 06cloud-services-team, 10Striker: Unread alerts should be much more visible - https://phabricator.wikimedia.org/T405526#11250026 (10Divyaratann) I am interested in this task so please confirm whether the repository is correct or not ? https://gerrit.wikimedia.org/r/admin/repos/labs/striker,general
[13:39:41] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
[13:40:27] <wmcs-alerts>	 RESOLVED: ToolsbetaNFSDown: No toolsbeta nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsbetaNFSDown
[13:46:27] <wmcs-alerts>	 FIRING: ToolsbetaNFSDown: No toolsbeta nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsbetaNFSDown
[13:51:27] <wmcs-alerts>	 RESOLVED: ToolsbetaNFSDown: No toolsbeta nfs services running found - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNFSDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsbetaNFSDown
[13:52:24] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
[13:52:41] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:58:02] <wikibugs>	 (03open) 10arian-ar: QoL improvements [toolforge-repos/wlm-iran] - 10https://gitlab.wikimedia.org/toolforge-repos/wlm-iran/-/merge_requests/2
[13:58:12] <wikibugs>	 (03merge) 10arian-ar: QoL improvements [toolforge-repos/wlm-iran] - 10https://gitlab.wikimedia.org/toolforge-repos/wlm-iran/-/merge_requests/2
[14:24:28] <wikibugs>	 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 06collaboration-services, 10GitLab (CI & Job Runners), 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11250330 (10Jelto)
[14:29:03] <wmcs-alerts>	 FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-65 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[14:36:31] <wikibugs>	 10VPS-project-devtools, 06collaboration-services, 10GitLab: Puppet failure on gitlab-1002.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T406234#11250458 (10dancy) Thanks @jelto!
[14:36:32] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
[14:36:35] <wikibugs>	 (03approved) 10dcaro: registry-admission: bump to 0.0.67-20251007080701-39739ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/990 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[14:36:39] <wikibugs>	 (03update) 10dcaro: registry-admission: bump to 0.0.67-20251007080701-39739ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/990 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[14:38:26] <wikibugs>	 (03merge) 10dcaro: registry-admission: bump to 0.0.67-20251007080701-39739ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/990 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[14:38:36] <wikibugs>	 (03update) 10dcaro: volume-admission: bump to 0.0.74-20251007080805-4d1a870d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/989 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[14:38:51] <wm-bot2>	 !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69
[14:38:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:39:33] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
[14:49:03] <wmcs-alerts>	 FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-65 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[14:51:03] <wm-bot2>	 !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69
[14:51:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:58:32] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[15:11:54] <wm-bot2>	 !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-69
[15:11:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:18:16] <wm-bot2>	 !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-69
[15:18:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:19:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[16:03:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-11 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[16:11:55] <wm-bot2>	 !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-11
[16:11:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:18:25] <wm-bot2>	 !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-11
[16:18:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:22:34] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
[16:23:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-11 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[16:27:43] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
[16:28:47] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11251074 (10dcaro)
[16:29:18] <wikibugs>	 (03approved) 10dcaro: volume-admission: bump to 0.0.74-20251007080805-4d1a870d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/989 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:29:20] <wikibugs>	 (03merge) 10dcaro: volume-admission: bump to 0.0.74-20251007080805-4d1a870d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/989 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:48:24] <wikibugs>	 06cloud-services-team, 10Striker: Unread alerts should be much more visible - https://phabricator.wikimedia.org/T405526#11251204 (10taavi) Noting here that I think {T332579} should happen before the alert bubble is made any more visible to prevent the feature becoming super annoying to those of us with hunders...
[17:05:56] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[17:08:58] <wikibugs>	 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 06collaboration-services, 10GitLab (CI & Job Runners), 13Patch-For-Review: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11251327 (10fnegri) a:05fnegri→03None I merged the patch above, but `reprepro` is fai...
[17:17:11] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[17:19:09] <wikibugs>	 10VPS-project-Codesearch, 06Wikipedia-iOS-App-Backlog: The Wikipedia iOS app should be added to Codesearch - https://phabricator.wikimedia.org/T406500#11251396 (10HNordeenWMF) p:05Triage→03Low
[17:58:15] <wikibugs>	 (03update) 10don-vip: Draft: Update to new NASA Photojournal [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/8
[18:00:25] <wikibugs>	 (03open) 10dduvall: CI: Specify syntax in blubber.yaml [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/43
[18:02:53] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Do something with cloudcontrol100[8-10]-dev - https://phabricator.wikimedia.org/T406630 (10Andrew) 03NEW
[18:10:34] <wikibugs>	 (03close) 10dduvall: CI: Specify syntax in blubber.yaml [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/43
[18:21:17] <wikibugs>	 (03open) 10dduvall: CI: Specify syntax in blubber.yaml [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/44
[18:22:58] <wikibugs>	 (03update) 10dduvall: CI: Specify syntax in blubber.yaml [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/44
[18:28:08] <wikibugs>	 (03approved) 10dcaro: CI: Specify syntax in blubber.yaml [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/44 (owner: 10dduvall)
[18:28:12] <wikibugs>	 (03merge) 10dcaro: CI: Specify syntax in blubber.yaml [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/44 (owner: 10dduvall)
[18:28:29] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[18:40:06] <wikibugs>	 10Cloud-VPS (Quota-requests), 06Release-Engineering-Team (Radar): Grant gitlab-runners-staging access to fast-iops volume type and a 4xiops instance flavor - https://phabricator.wikimedia.org/T406271#11251744 (10Andrew) ` root@cloudcontrol1011:~# openstack volume type show 2da5d224-a8e8-42ac-b5c6-3d234441d814...
[18:42:57] <wikibugs>	 06cloud-services-team, 10PAWS, 10OpenRefine: Unable to extract column from data for a specific file - https://phabricator.wikimedia.org/T406635 (10Perryprog) 03NEW
[18:43:28] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[18:48:35] <wikibugs>	 (03open) 10andrew: Gitlab runners staging [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/271
[18:50:15] <wikibugs>	 (03close) 10andrew: Gitlab runners staging [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/271
[18:51:07] <wikibugs>	 (03open) 10andrew: Gitlab runners staging [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/272
[18:51:11] <wikibugs>	 06cloud-services-team, 10PAWS, 10OpenRefine: Unable to extract column from data for a specific file - https://phabricator.wikimedia.org/T406635#11251788 (10Perryprog)
[18:51:25] <wikibugs>	 (03merge) 10andrew: Gitlab runners staging [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/272
[18:51:48] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
[18:52:05] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
[18:53:49] <wikibugs>	 (03open) 10andrew: Replace project ID with project name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/273
[18:53:57] <wikibugs>	 (03merge) 10andrew: Replace project ID with project name [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/273
[18:54:03] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
[18:54:24] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
[18:55:04] <wikibugs>	 06cloud-services-team, 10PAWS, 10OpenRefine: Unable to extract column from data for two specific files - https://phabricator.wikimedia.org/T406635#11251792 (10Perryprog)
[18:56:21] <logmsgbot_cloud>	 !log root@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
[18:56:22] <logmsgbot_cloud>	 !log root@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway
[18:56:33] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
[18:56:52] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
[18:57:01] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[18:57:27] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
[18:58:32] <jinxer-wm^>	 FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[18:59:20] <wikibugs>	 10Cloud-VPS (Quota-requests), 06Release-Engineering-Team (Radar): Grant gitlab-runners-staging access to fast-iops volume type and a 4xiops instance flavor - https://phabricator.wikimedia.org/T406271#11251793 (10Andrew) I added the flavor via tofu
[19:02:08] <wikibugs>	 06cloud-services-team, 10PAWS, 10OpenRefine: "Add columns from reconciled values" badly fails on images with no SDC at all - https://phabricator.wikimedia.org/T406635#11251808 (10Perryprog)
[19:24:02] <wikibugs>	 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11251891 (10Oreoluwa) Hello @Pepe_piton I would like to work on this task. When trying to set up the code I saw that I need to set up a basic Flask WSGI webservice. For me to set up I need to request membership for...
[19:46:47] <wikibugs>	 06cloud-services-team, 10PAWS, 10OpenRefine, 10Reconciliation: "Add columns from reconciled values" badly fails on images with no SDC at all - https://phabricator.wikimedia.org/T406635#11251964 (10LucasWerkmeister) Tagging #reconciliation since Abbe98 suggested the issue might be there rather than in OpenR...
[19:47:48] <jinxer-wm^>	 FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[20:16:48] <jinxer-wm^>	 FIRING: PuppetFailure: Puppet has failed on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[20:18:01] <wikibugs>	 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11252180 (10Pepe_piton) Hi @Oreoluwa! Thanks for joining the project! The requirement for a basic Flask WSGI webservice is only for those who want to set up Paulina on another production server. Outreachy contribut...
[20:21:48] <jinxer-wm^>	 FIRING: [2x] PuppetFailure: Puppet has failed on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[20:22:00] <wikibugs>	 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T406646 (10phaultfinder) 03NEW
[21:02:56] <jinxer-wm^>	 FIRING: [2x] SystemdUnitDown: The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[21:03:09] <wikibugs>	 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T406654 (10phaultfinder) 03NEW
[21:16:03] <wikibugs>	 (03approved) 10dcaro: jobs-api: bump to 0.0.423-20251007075945-b5b93756 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/988 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[21:16:11] <wikibugs>	 (03update) 10dcaro: jobs-api: bump to 0.0.423-20251007075945-b5b93756 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/988 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[21:17:00] <wikibugs>	 (03merge) 10dcaro: jobs-api: bump to 0.0.423-20251007075945-b5b93756 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/988 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[21:59:32] <wikibugs>	 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11252595 (10Sandijigs) Hi! I’m interested in working on this task. I’ve gone through the project description and the Wikibase REST API documentation, and I’d like to take this...
[22:03:30] <wikibugs>	 10Tool-paulina: Improve country dropdown for the "country of origin" filter in the advanced search for works. - https://phabricator.wikimedia.org/T405621#11252610 (10Sandijigs) Hi,I’d like to work on this improvement. I understand that the goal is to enable type-to-select functionality for the “Country of origin...
[22:07:08] <wikibugs>	 10Tool-paulina: Prefill nationality of author and country of origin of work at the country page - https://phabricator.wikimedia.org/T405604#11252628 (10Sandijigs) I’d like to take this issue. The task seems to involve prefilling the Nationality and Country of origin filters when viewing a country’s page, and sho...
[22:11:06] <wikibugs>	 10Tool-paulina: Improvements to the copyright term page: country list styles - https://phabricator.wikimedia.org/T405603#11252637 (10Sandijigs) I’d like to work on this feature. From what I understand, the goal is to replace the current long list of countries (with 50 and 70 years pma) with a grid layout that di...
[22:16:50] <wikibugs>	 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11252649 (10Sandijigs) I would  like to take this task. From what I understand, the goal is to display a short descriptive paragraph (sourced from the corresponding Wikidata item’s description) b...
[22:27:47] <wikibugs>	 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11252683 (10Nurah_Wakili) Hello @Pepe_piton , I would like to work on this task. I would like to kindly request to be added as a trusted contributor so I can fully participate in project tasks and updates. Thanks!
[22:49:54] <wikibugs>	 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11252740 (10Pepe_piton) a:03Sandijigs Great, @Sandijigs the task is assigned to you.  There's no specific format or style we prefer. You can simply focus on generating sampl...
[22:58:32] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[23:01:33] <wikibugs>	 10Tool-paulina: Clickable language menu options - https://phabricator.wikimedia.org/T402301#11252770 (10Nurah_Wakili) Hello, I would like to work on this task please.
[23:04:21] <wikibugs>	 10Tool-paulina: Proof of concept for editing statements in Paulina - https://phabricator.wikimedia.org/T405412#11252785 (10Nurah_Wakili) Hello @Pepe_piton , I would like to work on this task please.
[23:11:59] <wikibugs>	 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11252790 (10OlusegunFaj) Hello @Pepe_piton , I would like to work on this issue, can you please assign to me?
[23:20:24] <wikibugs>	 10Tool-paulina: Prefill nationality of author and country of origin of work at the country page - https://phabricator.wikimedia.org/T405604#11252806 (10AnbaNguyen) Hi! I'd like to work on this!
[23:20:46] <wikibugs>	 06cloud-services-team, 10Toolforge: sshd-session killed by Wheel of Misfortune on Toolforge bastion - https://phabricator.wikimedia.org/T406504#11252807 (10bd808) 05In progress→03Resolved
[23:24:21] <wikibugs>	 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11252834 (10AnbaNguyen) I'd like work on this @Pepe_piton
[23:24:38] <wikibugs>	 10Tool-paulina: Improvements to the copyright term page: country list styles - https://phabricator.wikimedia.org/T405603#11252836 (10Rolalove) Hi @Nat_WDU  I’d love to work on this task, could I please be assigned to it?
[23:43:30] <wikibugs>	 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598#11252852 (10Ronatone) If this task is still available, I would like to be assigned to it.
[23:48:03] <jinxer-wm>	 FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[23:51:44] <wikibugs>	 10Tool-paulina: Clickable language menu options - https://phabricator.wikimedia.org/T402301#11252874 (10Pepe_piton) a:03Nurah_Wakili Great, Nurah, it has been assigned to you.
[23:54:27] <wikibugs>	 10Tool-paulina: Minor improvements to the country page - https://phabricator.wikimedia.org/T405385#11252876 (10Pepe_piton) a:03Oreoluwa I assigned this task to @Oreoluwa.  @Nurah_Wakili I assigned this other task to you: T402301.  Thanks!
[23:58:08] <wikibugs>	 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11252881 (10Pepe_piton) a:03OlusegunFaj Sure @OlusegunFaj. I assigned it to you. Go ahead. Thanks!