[00:07:03] <wikibugs>	 (03PS1) 10Dzahn: add passwords::zuul::gerrit with fake password [labs/private] - 10https://gerrit.wikimedia.org/r/1174851 (https://phabricator.wikimedia.org/T395938)
[00:07:23] <wikibugs>	 (03PS2) 10Dzahn: add passwords::zuul::gerrit with fake password [labs/private] - 10https://gerrit.wikimedia.org/r/1174851 (https://phabricator.wikimedia.org/T395938)
[00:07:36] <wikibugs>	 (03CR) 10Dzahn: [V:03+2 C:03+2] add passwords::zuul::gerrit with fake password [labs/private] - 10https://gerrit.wikimedia.org/r/1174851 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn)
[02:54:08] <wikibugs>	 10Toolforge (Toolforge iteration 22): [components-api] store the config used for the deployment in the deployment themselves - https://phabricator.wikimedia.org/T400064#11052746 (10Raymond_Ndibe) a:03Raymond_Ndibe
[02:54:13] <wikibugs>	 10Toolforge (Toolforge iteration 22): [components-api] store the config used for the deployment in the deployment themselves - https://phabricator.wikimedia.org/T400064#11052749 (10Raymond_Ndibe) 05Open→03In progress
[02:58:40] <wikibugs>	 (03open) 10raymond-ndibe: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064)
[02:58:47] <wikibugs>	 (03update) 10raymond-ndibe: Draft: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064)
[02:58:51] <wikibugs>	 (03update) 10raymond-ndibe: Draft: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064)
[03:00:10] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.0.13 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/53
[03:00:15] <wikibugs>	 (03approved) 10raymond-ndibe: d/changelog: bump to 0.0.13 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/53
[03:00:22] <wikibugs>	 (03merge) 10raymond-ndibe: d/changelog: bump to 0.0.13 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/53
[03:04:24] <wikibugs>	 (03approved) 10raymond-ndibe: d/changelog: bump to 0.0.22 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/114
[03:04:25] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.0.22 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/114
[03:04:32] <wikibugs>	 (03merge) 10raymond-ndibe: d/changelog: bump to 0.0.22 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/114
[03:07:21] <wikibugs>	 (03update) 10raymond-ndibe: [cicd] replace bullseye with bookworm [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/46 (https://phabricator.wikimedia.org/T400616)
[03:07:24] <wikibugs>	 (03update) 10raymond-ndibe: [cicd] replace bullseye with bookworm [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/46 (https://phabricator.wikimedia.org/T400616)
[03:07:26] <wikibugs>	 (03approved) 10raymond-ndibe: [cicd] replace bullseye with bookworm [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/46 (https://phabricator.wikimedia.org/T400616)
[03:07:31] <wikibugs>	 (03merge) 10raymond-ndibe: [cicd] replace bullseye with bookworm [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/46 (https://phabricator.wikimedia.org/T400616)
[03:07:46] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.0.14 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/86 (https://phabricator.wikimedia.org/T363544 https://phabricator.wikimedia.org/T400616)
[03:07:49] <wikibugs>	 (03approved) 10raymond-ndibe: d/changelog: bump to 0.0.14 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/86 (https://phabricator.wikimedia.org/T363544 https://phabricator.wikimedia.org/T400616)
[03:14:49] <wikibugs>	 (03unapproved) 10raymond-ndibe: d/changelog: bump to 0.0.14 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/86 (https://phabricator.wikimedia.org/T363544 https://phabricator.wikimedia.org/T400616)
[03:24:21] <wikibugs>	 (03open) 10raymond-ndibe: d/changelog: bump to 0.3.6 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/47 (https://phabricator.wikimedia.org/T400616)
[03:25:12] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli
[03:36:05] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli
[03:37:51] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli
[03:48:16] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli
[03:49:57] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.3.6 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/47 (https://phabricator.wikimedia.org/T400616)
[03:49:59] <wikibugs>	 (03approved) 10raymond-ndibe: d/changelog: bump to 0.3.6 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/47 (https://phabricator.wikimedia.org/T400616)
[03:50:04] <wikibugs>	 (03merge) 10raymond-ndibe: d/changelog: bump to 0.3.6 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/47 (https://phabricator.wikimedia.org/T400616)
[04:12:28] <wmcs-alerts>	 FIRING: PuppetAgentFailure: Puppet agent failure detected on instance docker-registry-01 in project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[05:29:57] <wikibugs>	 06cloud-services-team: Request to enable XFF headers for test XTools hostnames - https://phabricator.wikimedia.org/T400964 (10MusikAnimal) 03NEW
[05:31:04] <wikibugs>	 06cloud-services-team: Request to enable XFF headers for test XTools hostnames - https://phabricator.wikimedia.org/T400964#11052821 (10MusikAnimal) I was interested to write a patch for this, but it seems the `instance-puppet` repo is [[ https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/ | hos...
[05:35:21] <wikibugs>	 06cloud-services-team: Request to enable XFF headers for test XTools hostnames - https://phabricator.wikimedia.org/T400964#11052826 (10MusikAnimal)
[05:39:33] <wmcs-alerts>	 FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesse
[06:27:08] <wikibugs>	 10Cloud-VPS (Quota-requests), 10XTools: Request increased quota for xtools Cloud VPS project - https://phabricator.wikimedia.org/T400853#11052855 (10MusikAnimal)
[06:31:29] <wmcs-alerts>	 FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 87.73%) at host paws-nfs-1 in project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull
[06:34:13] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/55
[06:34:22] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/113
[06:34:26] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/118
[06:34:28] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/118
[06:34:33] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/56
[06:34:39] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/56
[06:34:39] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/26
[06:34:48] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/22
[06:34:56] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/27
[06:34:57] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/26
[06:34:59] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/114
[06:35:15] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/22
[06:35:19] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/115
[06:35:21] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/27
[06:35:22] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/115
[06:35:25] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/114
[06:35:34] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/79
[06:35:37] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/79
[06:36:01] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/28
[06:36:04] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/28
[06:36:14] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/116
[06:36:17] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/87
[06:36:21] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/190
[06:36:22] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/190
[06:36:37] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/191
[06:36:38] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/116
[06:36:39] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/24
[06:36:44] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/87
[06:36:49] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/32
[06:36:49] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/88
[06:36:52] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/191
[06:36:57] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/24
[06:37:05] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/61
[06:37:05] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/32
[06:37:07] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/73
[06:37:17] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/138
[06:37:21] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/88
[06:37:25] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/48
[06:37:29] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/74
[06:37:33] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/259
[06:37:37] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/61
[06:37:42] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/73
[06:37:46] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/138
[06:37:50] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/51
[06:37:54] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/74
[06:37:58] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/49
[06:38:02] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/48
[06:38:06] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/259
[06:38:10] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/51
[06:38:14] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/49
[06:42:00] <wikibugs>	 (03open) 10bodhisattwa: Add uk.json [toolforge-repos/sangkalak] - 10https://gitlab.wikimedia.org/toolforge-repos/sangkalak/-/merge_requests/2
[07:09:04] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Request to enable XFF headers for test XTools hostnames - https://phabricator.wikimedia.org/T400964#11052901 (10taavi)
[08:04:19] <wikibugs>	 (03update) 10taavi: api: Allow querying logs for non-existent jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/189 (https://phabricator.wikimedia.org/T400913)
[08:23:21] <wm-bot2>	 Change on 12wikitech.wikimedia.org a page Help:Toolforge/Running jobs was modified, changed by Taavi-WMF link https://wikitech.wikimedia.org/w/index.php?diff=2329080 edit summary: /* Internal log storage */ rm some restrictions being removed due to the Loki move
[08:28:33] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Epic: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools - https://phabricator.wikimedia.org/T127367#11053090 (10taavi)
[08:34:15] <wm-bot2>	 Change on 12wikitech.wikimedia.org a page Help:Toolforge/Running jobs was modified, changed by Taavi-WMF link https://wikitech.wikimedia.org/w/index.php?diff=2329081 edit summary: /* Job logs */ note file log deprecation
[08:43:08] <wm-bot2>	 Change on 12wikitech.wikimedia.org a page Help:Toolforge/Running jobs was modified, changed by Taavi-WMF link https://wikitech.wikimedia.org/w/index.php?diff=2329082 edit summary: /* Job logs */ show internal log help first
[09:10:45] <wm-bot2>	 Change on 12wikitech.wikimedia.org a page News/2025 New Toolforge log storage was created, changed by Taavi-WMF link https://wikitech.wikimedia.org/wiki/News/2025_New_Toolforge_log_storage edit summary: init
[09:14:04] <wikibugs>	 06cloud-services-team, 10Toolforge: Check alerting for Loki - https://phabricator.wikimedia.org/T400981 (10taavi) 03NEW
[09:36:29] <wikibugs>	 06cloud-services-team, 10Toolforge: Check alerting for Loki - https://phabricator.wikimedia.org/T400981#11053224 (10taavi)
[09:36:34] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [o11y,logging,infra] Deploy Loki to store Toolforge tool log data - https://phabricator.wikimedia.org/T386480#11053225 (10taavi)
[09:39:31] <wm-bot2>	 Change on 12wikitech.wikimedia.org a page News/2025 New Toolforge log storage was modified, changed by Taavi-WMF link https://wikitech.wikimedia.org/w/index.php?diff=2329089 edit summary: expand
[09:40:31] <wikibugs>	 (03update) 10bodhisattwa: Add uk.json [toolforge-repos/sangkalak] - 10https://gitlab.wikimedia.org/toolforge-repos/sangkalak/-/merge_requests/2
[09:56:46] <wm-bot2>	 Change on 12wikitech.wikimedia.org a page Help:Toolforge/Running jobs was modified, changed by Taavi-WMF link https://wikitech.wikimedia.org/w/index.php?diff=2329092 edit summary: /* Job logs */ link to news page
[09:58:39] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Epic: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools - https://phabricator.wikimedia.org/T127367#11053244 (10taavi)
[10:00:30] <wikibugs>	 14Toolforge Build Service, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [tbs.cli] Create an easy way to extract/tail logs from buildpack based webservices - https://phabricator.wikimedia.org/T293672#11053248 (10taavi) →14Duplicate dup:03T336057
[10:00:39] <wikibugs>	 06cloud-services-team, 14Toolforge (Toolforge iteration 00), 13Patch-For-Review: Add commands to `webservice` and `jobs` to query logs from Kubernetes - https://phabricator.wikimedia.org/T336057#11053250 (10taavi)
[10:03:37] <wm-bot2>	 Change on 12wikitech.wikimedia.org a page News was modified, changed by Taavi-WMF link https://wikitech.wikimedia.org/w/index.php?diff=2329094 edit summary: [-17] refresh front page news
[10:54:13] <wikibugs>	 (03open) 10bodhisattwa: Add Ukrainian for interface language [toolforge-repos/sangkalak] - 10https://gitlab.wikimedia.org/toolforge-repos/sangkalak/-/merge_requests/3
[11:34:33] <wmcs-alerts>	 FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesse
[11:53:56] <wmcs-alerts>	 FIRING: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:58:56] <wmcs-alerts>	 RESOLVED: ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:46:28] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Add NEL headers to Cloud VPS and Toolforge managed proxies - https://phabricator.wikimedia.org/T400994 (10taavi) 03NEW
[12:46:35] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Add NEL headers to Cloud VPS and Toolforge managed proxies - https://phabricator.wikimedia.org/T400994#11053680 (10taavi) p:05Triage→03Low
[13:04:33] <wmcs-alerts>	 FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesse
[13:09:33] <wmcs-alerts>	 FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesse
[13:21:00] <wikibugs>	 (03merge) 10mahir256: Add Ukrainian for interface language [toolforge-repos/sangkalak] - 10https://gitlab.wikimedia.org/toolforge-repos/sangkalak/-/merge_requests/3 (owner: 10bodhisattwa)
[14:31:29] <wmcs-alerts>	 FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[14:39:58] <wmcs-alerts>	 RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance docker-registry-01 in project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[15:34:33] <wmcs-alerts>	 FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesse
[16:14:51] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:19:51] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:59:21] <wikibugs>	 06cloud-services-team, 10Toolforge: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014 (10LucasWerkmeister) 03NEW
[17:03:27] <wikibugs>	 10Tool-paulina: Greate functional tests for the basic flows - https://phabricator.wikimedia.org/T401015 (10DidiCoronel) 03NEW
[17:06:46] <wikibugs>	 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11054436 (10Nat_WDU) a:05Pepe_piton→03Nat_WDU
[17:10:22] <wikibugs>	 06cloud-services-team, 10Toolforge: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014#11054441 (10LucasWerkmeister) You can currently see the 403 at https://lucaswerkmeister-test.toolforge.org/hello.php (compare with https://lucaswerkmeister-test.toolforge.org/hello.html), th...
[17:10:54] <wikibugs>	 (03CR) 10Brouberol: [C:03+1] Add keytabs for new an-druid100[67] hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1171214 (https://phabricator.wikimedia.org/T397440) (owner: 10Stevemunene)
[17:16:31] <wikibugs>	 (03PS1) 10CDanis: devtools: add jenkins-deploy repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175150
[17:17:44] <wikibugs>	 (03PS2) 10CDanis: devtools: add jenkins-deploy repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175150
[17:42:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance docker-registry-01 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[17:42:39] <wikibugs>	 06cloud-services-team, 10Toolforge: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014#11054519 (10bd808) Let's see if we can figure out what is broken in the image. `lang=shell-session $ ssh dev.toolforge.org $ become bd808-test2 tools.bd808-test2@tools-bastion-12:~$ cat publ...
[17:57:28] <wmcs-alerts>	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance docker-registry-01 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[18:36:51] <wikibugs>	 (03open) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266)
[18:39:33] <wmcs-alerts>	 FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesse
[18:40:18] <wikibugs>	 (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266)
[18:56:11] <wikibugs>	 (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266)
[19:08:08] <wikibugs>	 (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266)
[19:36:27] <wikibugs>	 (03CR) 10Ladsgroup: [C:03+2] "It should be deployed in a day" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175150 (owner: 10CDanis)
[19:37:21] <wikibugs>	 (03Merged) 10jenkins-bot: devtools: add jenkins-deploy repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175150 (owner: 10CDanis)
[20:07:51] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[20:12:51] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[20:18:08] <wikibugs>	 06cloud-services-team, 10Toolforge: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014#11054855 (10bd808) The entry point of the Pod looks like this: `lang=yaml spec:   containers:                                                                        - command:     - /usr/bin...
[20:38:47] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge-static gives 301 redirects to backend server with port 8000 - https://phabricator.wikimedia.org/T401024 (10Bawolff) 03NEW
[21:01:43] <wikibugs>	 06cloud-services-team, 10Toolforge: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014#11054932 (10bd808) The argument passed to `--type` is `toolsws.backends.kubernetes.KubernetesBackend.get_types()["php7.3"]["cls"](tool, extra_args).name`. ``toolsws.backends.kubernetes.Kuber...
[21:03:03] <wikibugs>	 06cloud-services-team, 10Toolforge: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014#11054938 (10LucasWerkmeister) That would explain why the rolling restart still worked – the container spec (+ command) still stayed the same then.
[21:10:18] <wikibugs>	 (03open) 10bd808: lighttpd: Use "lighttpd" as webservice type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T401014)
[21:10:30] <wikibugs>	 (03update) 10bd808: lighttpd: Use "lighttpd" as webservice type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T401014)
[21:10:39] <wikibugs>	 (03update) 10bd808: lighttpd: Use "lighttpd" as webservice type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T401014)
[22:28:55] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Request to enable XFF headers for test XTools hostnames - https://phabricator.wikimedia.org/T400964#11055053 (10bd808) >>! In T400964#11052821, @MusikAnimal wrote: > I was interested to write a patch for this, but it seems the `instance-puppet` repo is [[ https://gerrit.wiki...
[23:52:27] <wikibugs>	 06cloud-services-team, 10Toolforge: Allow limiting exposed port access - https://phabricator.wikimedia.org/T400940#11055102 (10bd808) > Note: I do also use cross tool access, for example `cluebotng-review` using the IRC Relay; this could however be replaced with a local instance if a `only allow from same name...