[00:28:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-76 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [00:48:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [01:03:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [01:08:03] FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-14 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [02:03:03] FIRING: [5x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-14 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [03:28:03] FIRING: [6x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-14 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [04:28:03] FIRING: [6x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-14 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [04:48:03] FIRING: [6x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-14 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [06:46:29] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272376354 (https://github.com/cluebotng/component-configs/commits/449c7d3c68ade43ce238b9b335182dc50f929511) [06:46:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [06:49:45] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272390911 (https://github.com/cluebotng/component-configs/commits/49abbdd5dd7066314199c213043305ceed2b54f7) [06:49:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [07:39:56] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 [07:40:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:00:55] (03open) 10dcaro: use trixie [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/277 [08:18:32] !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 [08:18:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:19:00] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found [08:19:01] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found [08:19:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:19:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:28:48] RESOLVED: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-61 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:37:59] (03open) 10dcaro: image: use our hosted one in toolsbeta [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/10 [08:41:03] (03PS1) 10Adarsh2406: T406305: Update frontend build scripts and add Node/OpenSSL guidance [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1193785 (https://phabricator.wikimedia.org/T406305) [09:00:23] (03PS2) 10Adarsh2406: T406305: Update frontend build scripts and add Node/OpenSSL guidance [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1193785 (https://phabricator.wikimedia.org/T406305) [09:01:18] (03approved) 10dcaro: image: use our hosted one in toolsbeta [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/10 [09:01:21] (03merge) 10dcaro: image: use our hosted one in toolsbeta [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/10 [09:05:19] (03PS3) 10Adarsh2406: T406305: Update frontend build scripts and add Node/OpenSSL guidance [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1193785 (https://phabricator.wikimedia.org/T406305) [09:29:02] (03update) 10dcaro: use trixie [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/277 [09:41:04] (03update) 10dcaro: fetch minimal jobs [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/141 [09:46:24] (03update) 10dcaro: fetch minimal jobs [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/141 [09:47:13] (03update) 10dcaro: global: update generated toolforge models [repos/cloud/toolforge/components-api] (fetch_minimal_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/140 [09:47:22] (03update) 10dcaro: fetch minimal jobs [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/141 [09:48:08] (03update) 10dcaro: toolforge: get only the set job parameters [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/141 [09:54:33] (03open) 10dcaro: get_harbor.sh: use docker compose plugin, not docker-compose [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/75 [09:58:24] (03approved) 10dcaro: get_harbor.sh: use docker compose plugin, not docker-compose [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/75 [09:58:27] (03merge) 10dcaro: get_harbor.sh: use docker compose plugin, not docker-compose [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/75 [09:59:58] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.134-20251006095835-82d8efb6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/985 [10:06:47] (03PS1) 10Elukey: Add fake secrets for role::maps::master_bookworm [labs/private] - 10https://gerrit.wikimedia.org/r/1193812 [10:07:29] (03CR) 10Elukey: [V:03+2 C:03+2] Add fake secrets for role::maps::master_bookworm [labs/private] - 10https://gerrit.wikimedia.org/r/1193812 (owner: 10Elukey) [11:51:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-7 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:57:09] (03update) 10dcaro: use trixie [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/277 [11:59:49] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-7 [11:59:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:06:24] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-7 [12:06:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:11:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-7 has some processes stuck on NFS - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [12:36:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [13:01:28] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [13:16:11] 10Tool-paulina: Clickable language menu options - https://phabricator.wikimedia.org/T402301#11245985 (10Pepe_piton) a:05AkashKr_282→03None [13:24:30] (03update) 10dcaro: use trixie [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/277 [14:04:48] 10VPS-project-devtools, 06collaboration-services, 10GitLab: Puppet failure on gitlab-1002.devtools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T406234#11246223 (10Jelto) p:05Triage→03Medium a:03Jelto [14:08:26] (03update) 10dcaro: use trixie [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/277 [14:28:21] (03update) 10dcaro: use trixie [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/277 [14:42:58] (03update) 10dcaro: global: update generated toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/140 [14:43:27] (03close) 10dcaro: toolforge: get only the set job parameters [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/141 [15:16:01] 10VPS-project-Codesearch, 06Wikipedia-iOS-App-Backlog: The Wikipedia iOS app should be added to Codesearch - https://phabricator.wikimedia.org/T406500 (10jhsoby) 03NEW [15:27:56] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Create OAuth2 token in test Phabricator - https://phabricator.wikimedia.org/T406495#11246600 (10Aklapper) Note that the test instance currently has no external providers configured via https://phabricator.wmcloud.org... [15:28:02] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Create OAuth2 token in test Phabricator - https://phabricator.wikimedia.org/T406495#11246604 (10Aklapper) [15:30:03] 10VPS-project-Codesearch, 06Wikipedia-iOS-App-Backlog: The Wikipedia iOS app should be added to Codesearch - https://phabricator.wikimedia.org/T406500#11246611 (10Nemoralis) a:03Nemoralis [15:31:00] 10VPS-project-Codesearch, 06Wikipedia-Android-App-Backlog: Wikipedia Android app is not available on Codesearch - https://phabricator.wikimedia.org/T335407#11246615 (10Nemoralis) a:03Nemoralis [15:32:18] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Create OAuth2 token in test Phabricator - https://phabricator.wikimedia.org/T406495#11246619 (10Aklapper) See also {T377061}, though not directly related. [15:36:06] 10VPS-project-Codesearch, 06Wikipedia-Android-App-Backlog: Wikipedia Android app is not available on Codesearch - https://phabricator.wikimedia.org/T335407#11246630 (10Nemoralis) >>! In T335407#8815451, @Ladsgroup wrote: > I don't think we have a good section that would be able to hold it Do you think that we... [15:37:50] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations: Create OAuth2 token in test Phabricator - https://phabricator.wikimedia.org/T406495#11246633 (10LSobanski) There is https://idp.wmcloud.org, which is used for other projects (e.g. GitLab). [15:42:14] (03approved) 10fnegri: use trixie [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/277 (owner: 10dcaro) [15:51:28] 06cloud-services-team, 10Toolforge: sshd-session killed by Wheel of Misfortune on Toolforge bastion - https://phabricator.wikimedia.org/T406504 (10bd808) 03NEW [15:59:33] (03update) 10dcaro: global: update generated toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/140 [16:01:04] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [16:11:17] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: sshd-session killed by Wheel of Misfortune on Toolforge bastion - https://phabricator.wikimedia.org/T406504#11246830 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [16:25:39] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [16:26:31] (03merge) 10dcaro: use trixie [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/277 [16:36:14] (03update) 10dcaro: global: update generated toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/140 [16:38:46] (03PS3) 10Andrew Bogott: vps: Add cookbook to delete a project [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1139027 (https://phabricator.wikimedia.org/T391836) (owner: 10Majavah) [16:49:56] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: sshd-session killed by Wheel of Misfortune on Toolforge bastion - https://phabricator.wikimedia.org/T406504#11246997 (10Lucas_Werkmeister_WMDE) That would be [OpenSSH 9.8](https://www.openssh.com/releasenotes.html#9.8): > * sshd(8): the server has bee... [16:55:42] 10VPS-project-Codesearch, 06Wikipedia-iOS-App-Backlog: The Wikipedia iOS app should be added to Codesearch - https://phabricator.wikimedia.org/T406500#11247040 (10Dzahn) In the context of "official repos on GitHub" also see T405525 [16:57:03] 10VPS-project-Codesearch, 06Wikipedia-Android-App-Backlog: Wikipedia Android app is not available on Codesearch - https://phabricator.wikimedia.org/T335407#11247053 (10Dzahn) In the context of "official apps on github" also see T405525 [18:47:42] 06cloud-services-team: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516 (10Andrew) 03NEW [18:47:55] 06cloud-services-team: Upgrade openstack to version 'Flamingo' - https://phabricator.wikimedia.org/T406516#11247351 (10Andrew) [18:47:59] 06cloud-services-team, 10Horizon: Update our Horizon release to 2025.2 - https://phabricator.wikimedia.org/T405117#11247350 (10Andrew) [18:58:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:58:26] 06cloud-services-team: JobUnavailable Reduced availability for job openstack in cloud@codfw - https://phabricator.wikimedia.org/T406517 (10phaultfinder) 03NEW [19:23:42] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11247510 (10Nat_WDU) [19:24:34] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11247525 (10Nat_WDU) [19:36:39] 10Cloud-VPS (Quota-requests), 06Release-Engineering-Team (Radar): Grant gitlab-runners-staging access to fast-iops volume type and a 4xiops instance flavor - https://phabricator.wikimedia.org/T406271#11247592 (10dcaro) +1 [19:58:04] 06cloud-services-team: Package mcrouter for Debian Trixie - https://phabricator.wikimedia.org/T406522 (10Andrew) 03NEW [21:16:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [21:16:54] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcontrol2005-dev:9100 - https://phabricator.wikimedia.org/T406527 (10phaultfinder) 03NEW [22:58:32] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:23:18] (03PS1) 10Dzahn: add fake secret for zuul auth operator [labs/private] - 10https://gerrit.wikimedia.org/r/1193952 (https://phabricator.wikimedia.org/T395938) [23:24:04] (03CR) 10Dzahn: [V:03+2 C:03+2] add fake secret for zuul auth operator [labs/private] - 10https://gerrit.wikimedia.org/r/1193952 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn)