[00:41:06] (03PS1) 10Krinkle: index.php: Add link to Special:GlobalContributions for IP-address queries [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1132784 (https://phabricator.wikimedia.org/T382390) [00:41:26] (03CR) 10CI reject: [V:04-1] index.php: Add link to Special:GlobalContributions for IP-address queries [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1132784 (https://phabricator.wikimedia.org/T382390) (owner: 10Krinkle) [00:41:26] 10Tool-Global-user-contributions, 10CheckUser-GlobalContributions, 06Trust and Safety Product Team, 13Patch-For-Review: Add discovery link in GUC for temp user edits (to Special:GlobalContribs) - https://phabricator.wikimedia.org/T382390#10696550 (10Krinkle) Mock up: One IP: {F58956164} Multiple IPs: {... [00:45:54] (03update) 10ahecht: Draft: Cache database queries [toolforge-repos/afdstats] - 10https://gitlab.wikimedia.org/toolforge-repos/afdstats/-/merge_requests/3 [00:48:16] (03PS2) 10Krinkle: index.php: Add link to Special:GlobalContributions for IP-address queries [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1132784 (https://phabricator.wikimedia.org/T382390) [00:48:16] (03PS1) 10Krinkle: frontend: Add "notice" and "warning" styles [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1132785 [00:48:24] (03CR) 10Krinkle: [C:03+2] Main: Rename "isIp" to "isSingleIp" for clarity [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1132774 (owner: 10Krinkle) [00:48:28] (03CR) 10Krinkle: [C:03+2] frontend: Add "notice" and "warning" styles [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1132785 (owner: 10Krinkle) [00:48:59] (03Merged) 10jenkins-bot: Main: Rename "isIp" to "isSingleIp" for clarity [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1132774 (owner: 10Krinkle) [00:49:00] (03Merged) 10jenkins-bot: frontend: Add "notice" and "warning" styles [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1132785 (owner: 10Krinkle) [04:36:59] 06cloud-services-team, 10Tool-suggestbotbn, 10Toolforge: Ssh login to `login.toolforge.org` failing for uid=shohag - https://phabricator.wikimedia.org/T390614#10697112 (10ShohagS) >>! In T390614#10695593, @taavi wrote: > Can you run ssh with the `-v` flag included and paste the output here? ╭─shohag@Shohag-... [04:39:07] 06cloud-services-team, 10Tool-suggestbotbn, 10Toolforge: Ssh login to `login.toolforge.org` failing for uid=shohag - https://phabricator.wikimedia.org/T390614#10697113 (10ShohagS) >>! In T390614#10696391, @bd808 wrote: > A comment on https://serverfault.com/questions/813542/ssh-fork-of-unprivileged-child-fai... [04:41:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-48 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [06:23:16] 10Cloud-VPS (Quota-requests): Increase disk quota for math - https://phabricator.wikimedia.org/T390656 (10Physikerwelt) 03NEW [06:31:39] 10Cloud-VPS (Quota-requests): Increase disk quota for math - https://phabricator.wikimedia.org/T390656#10697201 (10Physikerwelt) While the overall compressed size of all WMF-hosted projects is only 1.5GB decompression and rendering needs additional space. 50GB is quite an realistic estimate I think. In theory,... [06:33:50] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/26 [06:33:53] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/67 [06:33:59] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/93 [06:34:00] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/65 [06:34:03] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/67 [06:34:04] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/93 [06:34:14] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/65 [06:34:14] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/22 [06:34:24] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/19 [06:34:27] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/22 [06:34:27] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/19 [06:34:36] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/19 [06:34:39] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/102 [06:34:43] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/19 [06:34:47] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/102 [06:34:51] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/73 [06:34:55] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/73 [06:35:02] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/151 [06:35:06] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/77 [06:35:07] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/151 [06:35:12] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/77 [06:35:16] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/20 [06:35:20] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/20 [06:35:24] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/28 [06:35:28] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/28 [06:35:39] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/127 [06:35:42] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/62 [06:35:44] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/127 [06:35:46] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/62 [06:35:50] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/234 [06:35:54] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/69 [06:35:58] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/69 [06:36:02] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/46 [06:36:06] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/234 [06:36:10] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/46 [06:36:14] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/41 [06:36:18] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/41 [06:56:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-48 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [06:57:00] 10VPS-project-Wikistats, 07affects-Miraheze: wikistats - import miraheze timer/service failed - https://phabricator.wikimedia.org/T390593#10697260 (10RhinosF1) >>! In T390593#10695656, @Dzahn wrote: > Thank you! The service runs again. > > regarding the API change: Maybe it could be reconsidered if renaming... [06:57:13] 10VPS-project-Wikistats, 07affects-Miraheze: wikistats - import miraheze timer/service failed - https://phabricator.wikimedia.org/T390593#10697261 (10RhinosF1) 05Open→03Resolved a:03Paladox [08:29:15] 06cloud-services-team, 10Tool-suggestbotbn, 10Toolforge: Ssh login to `login.toolforge.org` failing for uid=shohag - https://phabricator.wikimedia.org/T390614#10697451 (10taavi) The VS Code remote SSH connection uses [[ https://code.visualstudio.com/docs/remote/vscode-server | VS Code Server ]] which is non-... [09:07:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-48 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [09:16:31] 06cloud-services-team, 10Tool-suggestbotbn, 10Toolforge: Ssh login to `login.toolforge.org` failing for uid=shohag - https://phabricator.wikimedia.org/T390614#10697566 (10ShohagS) >>! In T390614#10697451, @taavi wrote: > The VS Code remote SSH connection uses [[ https://code.visualstudio.com/docs/remote/vsco... [09:36:29] 06cloud-services-team, 10Tool-suggestbotbn, 10Toolforge: Ssh login to `login.toolforge.org` failing for uid=shohag - https://phabricator.wikimedia.org/T390614#10697635 (10taavi) I did kill those processes already. [09:48:22] 06cloud-services-team, 10Cloud-VPS: openstack: fix missing prometheus metrics - https://phabricator.wikimedia.org/T373878#10697689 (10fnegri) Nice :) [11:17:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-45 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [11:20:59] !log aborrero@cloudcumin1001 dwl START - Cookbook wmcs.openstack.quota_increase (T389711) [11:21:03] T389711: Add new flavor for dwl project and increase quota - https://phabricator.wikimedia.org/T389711 [11:21:07] !log aborrero@cloudcumin1001 dwl END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T389711) [11:22:25] 10Cloud-VPS (Quota-requests): Add new flavor for dwl project and increase quota - https://phabricator.wikimedia.org/T389711#10698037 (10aborrero) 05Open→03Resolved a:03aborrero [11:25:07] 10Tool-fault-tolerance: Fault-tolerance tool should have a backend option - https://phabricator.wikimedia.org/T389612#10698047 (10Ladsgroup) Would that work for you? [11:32:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-45 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [11:47:37] 10Tool-fault-tolerance: Fault-tolerance tool should have a backend option - https://phabricator.wikimedia.org/T389612#10698112 (10MatthewVernon) Yes, for my purposes, considering e.g. thanos-be* as one cluster would be entirely fine. Sorry, I meant to reply to your previous. [11:57:52] (03open) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [11:59:30] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [12:00:28] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [12:03:45] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/13 (owner: 10l10n-bot) [12:03:47] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/13 (owner: 10l10n-bot) [12:07:44] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [12:08:39] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [12:38:22] (03approved) 10fnegri: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) (owner: 10aborrero) [12:41:16] (03unapproved) 10fnegri: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) (owner: 10aborrero) [12:48:28] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Upgrade "toolsbeta" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390212#10698650 (10fnegri) [12:48:48] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Upgrade "toolsbeta" cluster to k8s 1.29.15 - https://phabricator.wikimedia.org/T390212#10698652 (10fnegri) 05In progress→03Resolved [12:49:40] 10Cloud-VPS (Quota-requests): Add new flavor for dwl project and increase quota - https://phabricator.wikimedia.org/T389711#10698657 (10Giftpflanze) Thank you for the ample RAM increase. I do not have access to the requested flavor, though. Did you miss the request/was it denied/did you forget to create it o... [12:57:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-45 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [13:02:03] FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-24 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [13:42:08] 06cloud-services-team, 10PAWS: Duplicate eqiad1 PAWS setup in codfw1dev - https://phabricator.wikimedia.org/T390726 (10Andrew) 03NEW [13:49:48] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 [13:54:33] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 [13:55:10] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 [13:56:16] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 [13:59:55] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 [14:01:38] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 [14:07:23] 10Cloud-VPS (Quota-requests): Add new flavor for dwl project and increase quota - https://phabricator.wikimedia.org/T389711#10699285 (10aborrero) 05Resolved→03Open Sorry, I got caught in a rabbit hole of refactoring the code required to maintain the flavors. I'll do shortly. [14:10:10] (03open) 10andrew: Add paws project in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/164 (https://phabricator.wikimedia.org/T390726) [14:12:03] FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-24 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [14:14:28] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): Upgrade "tools" cluster to k8s 1.29 - https://phabricator.wikimedia.org/T390214#10699302 (10fnegri) 05Open→03In progress [14:17:18] FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-24 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [14:22:03] FIRING: [4x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-24 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [14:27:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-41 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [14:28:48] (03open) 10aborrero: flavors: add new g4.cores16.ram64.disk20 flavor for dwl project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/165 (https://phabricator.wikimedia.org/T389711) [14:31:39] (03update) 10aborrero: flavors: add new g4.cores16.ram64.disk20 flavor for dwl project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/165 (https://phabricator.wikimedia.org/T389711) [14:32:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-41 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [14:33:34] (03close) 10andrew: Add paws project in codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/164 (https://phabricator.wikimedia.org/T390726) [14:49:52] (03merge) 10aborrero: flavors: add new g4.cores16.ram64.disk20 flavor for dwl project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/165 (https://phabricator.wikimedia.org/T389711) [14:50:03] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:50:49] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:58:45] 06cloud-services-team, 10PAWS, 13Patch-For-Review: Duplicate eqiad1 PAWS setup in codfw1dev - https://phabricator.wikimedia.org/T390726#10699558 (10Andrew) oops, we already have a project, pawsdev. I just wasn't in it. [15:01:04] 10VPS-project-Wikistats, 07affects-Miraheze: wikistats - import miraheze timer/service failed - https://phabricator.wikimedia.org/T390593#10699568 (10Dzahn) Gotcha! Thanks. Let's get to that once it's merged upstream then. [15:10:16] (03approved) 10lucaswerkmeister-wmde: Lint the repository, and add CI job configration to validate PRs [toolforge-repos/phpunit-results-cache] (feat/merge-cache-data-20250328) - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/7 (https://phabricator.wikimedia.org/T388767) (owner: 10arthurtaylor) [15:10:50] (03merge) 10lucaswerkmeister-wmde: Lint the repository, and add CI job configration to validate PRs [toolforge-repos/phpunit-results-cache] (feat/merge-cache-data-20250328) - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/7 (https://phabricator.wikimedia.org/T388767) (owner: 10arthurtaylor) [15:10:51] (03update) 10lucaswerkmeister-wmde: Merge cache data from independent test runs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/6 (https://phabricator.wikimedia.org/T388767) (owner: 10arthurtaylor) [15:15:10] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [15:26:08] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [15:27:30] 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Add new flavor for dwl project and increase quota - https://phabricator.wikimedia.org/T389711#10699761 (10aborrero) 05Open→03Resolved @Giftpflanze you should be all set now. [15:28:07] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [15:40:09] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [15:44:55] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [15:47:54] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [15:55:13] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [16:47:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-48 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [16:52:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-48 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [17:10:14] 10Tool-fa-speed, 06Future-Audiences: [2025 Apr 01-Apr11] Committed sprint work - https://phabricator.wikimedia.org/T390687#10700291 (10Maryana) [18:05:23] 06cloud-services-team, 10Data-Services: Remove the compatability layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767 (10Ladsgroup) 03NEW [19:53:37] 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10700999 (10Andrew) Thanks for the look! Since my goal here is 'adequate in an emergency' I'm going to set aside many of your concerns as aesthetic... [21:00:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [21:00:52] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch [21:01:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [21:02:01] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [21:12:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,designate [21:12:20] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,designate [21:38:28] 10VPS-project-Wikistats, 07affects-Miraheze: wikistats - import miraheze timer/service failed - https://phabricator.wikimedia.org/T390593#10701514 (10Dzahn) Thank you @Universal_Omega for the patch at https://gitlab.wikimedia.org/cloudvps-repos/wikistats/-/merge_requests/11 and thanks RhinosF1 for review/... [22:27:01] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: [Session] Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10701633 (10debt) Hello - we've published the program schedule: https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2025/Program#S... [23:15:26] 06cloud-services-team, 10Cloud-VPS: Enable use of web proxy for wikipeoplestats.org domain - https://phabricator.wikimedia.org/T390800 (10Danielyepezgarces) 03NEW