[00:10:27] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [00:10:46] (03update) 10raymond-ndibe: [components-smoke-test] reduce testcase dep on each other [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:12:31] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [00:13:32] (03update) 10raymond-ndibe: [components-smoke-test] reduce testcase dep on each other [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:14:19] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:14:41] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:14:55] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:18:02] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:19:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-55 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [00:19:28] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:19:37] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [00:20:12] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:27:31] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:27:47] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [00:29:47] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:30:23] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [00:31:47] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:32:30] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:33:18] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [00:35:06] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [00:35:57] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:48:33] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [00:52:09] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:52:59] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [00:56:07] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [00:56:12] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [00:58:33] (03update) 10raymond-ndibe: [deploy] skip build if refs are same [repos/cloud/toolforge/components-api] (fix_do_run_image_name_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044) [00:59:05] (03update) 10raymond-ndibe: [deploy] add force-build and force-run query params [repos/cloud/toolforge/components-api] (skip_build_if_refs_are_same) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/80 (https://phabricator.wikimedia.org/T389044) [01:01:54] (03update) 10raymond-ndibe: [deploy] skip build if refs are same [repos/cloud/toolforge/components-api] (fix_do_run_image_name_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044) [01:02:00] (03update) 10raymond-ndibe: [deploy] skip build if refs are same [repos/cloud/toolforge/components-api] (fix_do_run_image_name_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044) [01:02:00] (03approved) 10raymond-ndibe: [deploy] skip build if refs are same [repos/cloud/toolforge/components-api] (fix_do_run_image_name_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044) [01:19:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-55 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [01:22:37] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [01:22:50] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [01:23:12] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [01:23:17] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [01:30:13] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [01:30:40] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [01:30:50] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [01:31:09] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [03:00:56] (03update) 10raymond-ndibe: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [03:01:59] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [03:04:27] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [03:15:49] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [03:31:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-55 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [03:46:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-55 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [04:01:13] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [04:06:42] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [04:21:00] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [06:48:05] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-55 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [06:58:04] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-55 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:08:34] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10882664 (10Gehel) [07:44:02] (03PS6) 10Slyngshede: Build: Update build system [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1143806 [07:44:37] (03CR) 10CI reject: [V:04-1] Build: Update build system [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1143806 (owner: 10Slyngshede) [07:53:18] (03PS7) 10Slyngshede: Build: Update build system [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1143806 [07:53:54] (03CR) 10CI reject: [V:04-1] Build: Update build system [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1143806 (owner: 10Slyngshede) [07:56:40] (03CR) 10Slyngshede: "Task for updating Blubber is available here: https://phabricator.wikimedia.org/T395645 and there's already a patch for review." [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1143806 (owner: 10Slyngshede) [07:57:24] (03CR) 10Slyngshede: [C:04-1] "Still need to fix the missing logging." [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1143806 (owner: 10Slyngshede) [08:02:10] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882766 (10Marostegui) We are going to start with s6 [08:50:55] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10882901 (10Marostegui) [08:51:07] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10882904 (10Marostegui) clouddb1020:x3 is now up [08:57:46] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882911 (10fnegri) [09:09:04] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882935 (10fnegri) [09:12:59] 06cloud-services-team, 10Data-Services: maintain-dbusers: Cleanup rows for old hosts - https://phabricator.wikimedia.org/T395993 (10taavi) 03NEW p:05Triage→03Medium [09:12:59] 06cloud-services-team, 10Data-Services: maintain-dbusers: Cleanup rows for old hosts - https://phabricator.wikimedia.org/T395994 (10taavi) 03NEW p:05Triage→03Medium [09:13:37] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882975 (10fnegri) > We are going to start with s6 Ok, then I will start with clouddb1015 and clouddb1019 w... [09:17:54] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10882978 (10Marostegui) The steps look good to me, thank you [09:19:04] 06cloud-services-team, 10Toolforge, 10ChangeProp, 06collaboration-services, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10882980 (10hashar) 05Open→03Resolved a:03hashar After chatting with Alexandros, the relice... [09:34:51] 10Data-Services, 06DBA: Remove sanitarium hosts from codfw - https://phabricator.wikimedia.org/T394884#10883022 (10Marostegui) 05Open→03Resolved This is done [09:42:34] (03PS3) 10Majavah: build: Add gbp.conf with id-length [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153344 [09:42:34] (03PS1) 10Majavah: sql: Add --extension to tab completion [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153567 [09:43:59] (03CR) 10CI reject: [V:04-1] build: Add gbp.conf with id-length [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153344 (owner: 10Majavah) [10:23:08] 06cloud-services-team, 10Toolforge: Toolforge `sql` utility should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395901#10883159 (10taavi) p:05Triage→03Medium [10:34:43] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10883193 (10BTullis) >>! In T390954#10879551, @Marostegui wrote: >>>! In T390954#10879549, @BTullis wrote: >> @Marostegui - Thanks. I'll double-check wi... [10:51:07] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10883241 (10taavi) [10:51:43] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10883242 (10Marostegui) >>! In T390954#10883193, @BTullis wrote: >>>! In T390954#10879551, @Marostegui wrote: >>>>! In T390954#10879549, @BTullis wrote:... [10:52:04] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10883243 (10Marostegui) 05Open→03Resolved So I think this is all done then. Replication is running on clouddb1016 and 1020. [10:53:58] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10883248 (10taavi) Indeed, thanks all! [10:55:06] (03CR) 10Majavah: [C:03+2] sql: Add --extension to tab completion [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153567 (owner: 10Majavah) [10:55:37] (03Merged) 10jenkins-bot: sql: Add --extension to tab completion [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153567 (owner: 10Majavah) [10:55:56] (03CR) 10Majavah: [V:03+2 C:03+2] build: Add gbp.conf with id-length [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153344 (owner: 10Majavah) [10:56:42] (03PS1) 10Majavah: Add changelog entry for 1.49 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153580 [10:57:26] (03CR) 10Majavah: [V:03+2 C:03+2] Add changelog entry for 1.49 [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153580 (owner: 10Majavah) [10:59:36] 06cloud-services-team, 10Toolforge: wmcs-package-build syntax warning - https://phabricator.wikimedia.org/T396004 (10taavi) 03NEW [10:59:43] 06cloud-services-team, 10Data-Services: maintain-dbusers: Cleanup rows for old hosts - https://phabricator.wikimedia.org/T395994#10883266 (10Aklapper) →14Duplicate dup:03T395993 [10:59:44] 06cloud-services-team, 10Data-Services: maintain-dbusers: Cleanup rows for old hosts - https://phabricator.wikimedia.org/T395993#10883268 (10Aklapper) [11:14:06] 10Cloud-VPS (Quota-requests), 06collaboration-services: Request to increase volume quota in devtools - https://phabricator.wikimedia.org/T396006 (10Jelto) 03NEW [11:17:42] (03PS1) 10Majavah: Fix distro for 1.49 Debian release [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153585 [11:17:57] (03CR) 10Majavah: [V:03+2 C:03+2] Fix distro for 1.49 Debian release [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153585 (owner: 10Majavah) [11:20:26] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: wmcs-package-build syntax warning - https://phabricator.wikimedia.org/T396004#10883333 (10taavi) a:03taavi [11:20:35] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: wmcs-package-build syntax warning - https://phabricator.wikimedia.org/T396004#10883334 (10taavi) p:05Triage→03Low [11:34:47] (03CR) 10Majavah: toolforge.deploy: support deploying packages (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 (owner: 10David Caro) [11:38:13] 06cloud-services-team, 10Toolforge: Toolforge `sql` utility should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395901#10883371 (10taavi) 05Open→03Resolved [11:39:54] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcephosd200[567] - https://phabricator.wikimedia.org/T393614#10883376 (10cmooney) >>! In T393614#10881864, @Jhancock.wm wrote: > yeah same rack. I just need someone to migrate it for me since cloud is more complex tha... [11:49:38] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10883414 (10cmooney) >>! In T394333#10881532, @Andrew wrote: > Among other things, the connection speed for 1048 looks pretty wrong; we were hoping this wo... [12:06:41] 10Tool-openstack-browser: OpenStack Browser project view broken - https://phabricator.wikimedia.org/T396011 (10LucasWerkmeister) 03NEW [12:07:16] 10Tool-openstack-browser: OpenStack Browser project view broken - https://phabricator.wikimedia.org/T396011#10883489 (10taavi) p:05Triage→03High a:03taavi [12:08:20] 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps openstack to version 'Epoxy' - https://phabricator.wikimedia.org/T390914#10883491 (10taavi) [12:08:23] 10Tool-openstack-browser: OpenStack Browser project view broken - https://phabricator.wikimedia.org/T396011#10883492 (10taavi) [12:09:15] 10Tool-openstack-browser: OpenStack Browser project view broken - https://phabricator.wikimedia.org/T396011#10883493 (10taavi) 05Open→03Resolved Fixed with https://gitlab.wikimedia.org/toolforge-repos/openstack-browser/-/commit/62469865efa00c040fe11b02794bacae95343fb5. [12:28:04] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10883615 (10cmooney) >>! In T393102#10876497, @Jhancock.wm wrote: > @Andrew not sure why but i can't get it to pxe at all anymore. Can you take a look for m... [12:50:59] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10883701 (10fnegri) 05In progress→03Resolved [13:12:23] 06cloud-services-team, 10Cloud-VPS: Review our handling of keystone 'member' role (previously known as 'projectadmin') - https://phabricator.wikimedia.org/T396016 (10Andrew) 03NEW [13:37:04] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-55 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:47:04] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-55 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [14:25:17] (03update) 10taavi: Remove toolsbeta-prometheus-1 volume [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/45 [14:29:09] 06cloud-services-team, 10Toolforge: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10884224 (10Nokib_Sarkar) 05In progress→03Resolved a:03Nokib_Sarkar [14:31:27] 10Tool-campwiz-nxt: Implement Reverse proxy and Failover server into campwiz nxt - https://phabricator.wikimedia.org/T394730#10884237 (10Nokib_Sarkar) 05In progress→03Resolved a:03Nokib_Sarkar [14:34:28] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcephosd200[567] - https://phabricator.wikimedia.org/T393614#10884260 (10Jhancock.wm) Those are the new servers that i'm still getting racked. The one i need to move a port for is cloudcephosd2003-dev, connected to po... [14:35:49] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10884262 (10Jhancock.wm) oh whoops. ty! [14:37:16] 06cloud-services-team, 10Horizon, 06Security-Team, 07SecTeam-Processed, and 2 others: OpenStack / Horizon lets “reader” users promote anyone to “member” - https://phabricator.wikimedia.org/T396013#10884275 (10sbassett) [14:46:14] 06cloud-services-team, 10Toolforge: [infra] Reports of slow connectivity from APAC - https://phabricator.wikimedia.org/T395135#10884319 (10Nokib_Sarkar) From Dhaka ##What is the output of http://test-ipv6.com/helpdesk/ ? ` Help desk code: 4 IPv4 Only IPv4: Good, AS132298 - ADCL-AS-AP Antaranga Dot Com Ltd IPv... [14:54:29] 06cloud-services-team, 10Toolforge: Update maintain_kubeusers to use the toolstate database - https://phabricator.wikimedia.org/T334629#10884349 (10Raymond_Ndibe) a:03Raymond_Ndibe [14:56:04] 10Cloud-Services, 06cloud-services-team: toolsbeta paging - https://phabricator.wikimedia.org/T396038 (10Andrew) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project... [14:56:43] 06cloud-services-team, 10Toolsbeta-Tools: toolsbeta paging - https://phabricator.wikimedia.org/T396038#10884370 (10Andrew) [14:58:35] 06cloud-services-team, 10Toolforge: toolsbeta paging - https://phabricator.wikimedia.org/T396038#10884387 (10taavi) [15:31:41] FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [15:37:53] (03CR) 10Beleke Ian: [C:03+1] rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [15:46:41] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [15:56:41] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [16:06:41] RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [16:08:28] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10884693 (10VirginiaPoundstone) @fnegri can you remind me what initiative this supports? Better yet add the context to... [16:25:42] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06collaboration-services: Request to increase volume quota in devtools - https://phabricator.wikimedia.org/T396006#10884860 (10bd808) +1 [16:28:29] 10PAWS: /home/paws is 100% - https://phabricator.wikimedia.org/T396051 (10jeremyb-phone) 03NEW [16:28:42] 10PAWS: /home/paws is 100% - https://phabricator.wikimedia.org/T396051#10884901 (10jeremyb-phone) p:05Triage→03Unbreak! [17:01:16] 10PAWS: /home/paws is 100% - https://phabricator.wikimedia.org/T396051#10885044 (10dcaro) p:05Unbreak!→03High Just did a quick remove of 'files-to-remove', that freed a little bit of space, enough to let some time for someone to cleanup in depth tomorrow/later [17:03:08] 10PAWS: /home/paws is 100% - https://phabricator.wikimedia.org/T396051#10885050 (10dcaro) Maybe @Andrew can give it a quick look today if you are working? [17:59:30] (03PS1) 10Andrew Bogott: Another fix for directing proxies to Octavia LBs [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153669 [18:00:03] (03CR) 10CI reject: [V:04-1] Another fix for directing proxies to Octavia LBs [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153669 (owner: 10Andrew Bogott) [18:00:31] (03PS2) 10Andrew Bogott: Another fix for directing proxies to Octavia LBs [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153669 [18:01:00] (03CR) 10CI reject: [V:04-1] Another fix for directing proxies to Octavia LBs [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153669 (owner: 10Andrew Bogott) [18:02:48] (03PS3) 10Andrew Bogott: Another fix for directing proxies to Octavia LBs [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153669 [18:07:33] (03CR) 10Andrew Bogott: [C:03+2] Another fix for directing proxies to Octavia LBs [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153669 (owner: 10Andrew Bogott) [18:08:03] (03Merged) 10jenkins-bot: Another fix for directing proxies to Octavia LBs [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153669 (owner: 10Andrew Bogott) [18:22:15] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: cloudcontrol2010-dev service implementation - https://phabricator.wikimedia.org/T396064 (10Andrew) 03NEW [18:22:42] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10885418 (10Andrew) [18:22:57] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10885419 (10Andrew) 05Open→03Resolved This is now imaged and ready for service. [18:46:53] 10PAWS: /home/paws is 100% - https://phabricator.wikimedia.org/T396051#10885467 (10Andrew) I freed up a small amount of space. The docs say: > There is a job run to clean user accounts down to 5gb once a day. Files removed from user directories are placed in paws-nfs-1.paws.eqiad1.wikimedia.cloud:/srv/paws/fi... [18:48:24] 10PAWS: /home/paws is 100% - https://phabricator.wikimedia.org/T396051#10885469 (10Andrew) If it is running, or is revived, we should probably move from 5G to 4, or expand the nfs volume. [20:02:58] 10PAWS: /home/paws is 100% - https://phabricator.wikimedia.org/T396051#10885707 (10Corvid4444) The abusive accounts had a tendency to install freeroot which has the unfortunate effect of pulling down gigabytes of small files. The shrink script shrinks by removing the largest file one at a time until the home dir... [20:49:43] !log raymond-ndibe@cloudcumin1001 devtools START - Cookbook wmcs.openstack.quota_increase (T396006) [20:49:48] T396006: Request to increase volume quota in devtools - https://phabricator.wikimedia.org/T396006 [20:49:50] !log raymond-ndibe@cloudcumin1001 devtools END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T396006) [20:52:46] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06collaboration-services: Request to increase volume quota in devtools - https://phabricator.wikimedia.org/T396006#10885858 (10Raymond_Ndibe) **Old:** ` raymond-ndibe@cloudcontrol1006:~$ sudo wmcs-openstack quota show devtools +-----------------------+----... [20:53:15] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06collaboration-services: Request to increase volume quota in devtools - https://phabricator.wikimedia.org/T396006#10885859 (10Raymond_Ndibe) 05Open→03Resolved [20:58:30] 10Cloud-VPS (Quota-requests), 06collaboration-services: Request to increase RAM and VCPU quota in codesearch - https://phabricator.wikimedia.org/T396073 (10Dzahn) 03NEW [21:04:49] 10Cloud-VPS (Quota-requests): Pixel project "disk40" flavor, and perhaps a few more cores? - https://phabricator.wikimedia.org/T395837#10885894 (10Raymond_Ndibe) Hello @Mhurd can you structure you request a bit like the example below to enable us understand exactly what you are asking for: ` Project Name: (03CR) 10Eugene233: [C:03+2] rearrange the location of some files [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [21:22:27] (03CR) 10Eugene233: [C:03+2] added endpoints.ts [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1151745 (owner: 10Martindevelops) [21:23:33] (03CR) 10Eugene233: "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1151745 (owner: 10Martindevelops) [21:23:57] (03CR) 10Eugene233: [C:03+2] "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1151745 (owner: 10Martindevelops) [21:24:06] (03CR) 10Eugene233: [C:03+2] "recheck" [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1152117 (owner: 10NkwadaNora) [21:28:27] (03PS1) 10Krinkle: write_config: Add jsdoc/wmf-theme repo to Codesearch [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1153714 [21:28:36] (03CR) 10Krinkle: [C:03+2] write_config: Add jsdoc/wmf-theme repo to Codesearch [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1153714 (owner: 10Krinkle) [21:31:56] (03Merged) 10jenkins-bot: write_config: Add jsdoc/wmf-theme repo to Codesearch [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1153714 (owner: 10Krinkle) [22:32:34] (03PS1) 10Andrew Bogott: Rework the way we gather the loadbalancer list [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153732 [22:33:03] (03CR) 10CI reject: [V:04-1] Rework the way we gather the loadbalancer list [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153732 (owner: 10Andrew Bogott) [22:35:12] (03PS2) 10Andrew Bogott: Rework the way we gather the loadbalancer list [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153732 [22:36:09] (03CR) 10Andrew Bogott: [C:03+2] Rework the way we gather the loadbalancer list [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1153732 (owner: 10Andrew Bogott) [22:52:14] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06collaboration-services: Request to increase RAM and VCPU quota in codesearch - https://phabricator.wikimedia.org/T396073#10886116 (10bd808) +1 [22:52:51] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:57:51] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:59:19] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Test (and implement?) Openstack Octavia lbaas - https://phabricator.wikimedia.org/T393783#10886148 (10Andrew) 05Open→03Resolved [23:20:35] !log raymond-ndibe@cloudcumin1001 codesearch START - Cookbook wmcs.openstack.quota_increase (T396073) [23:20:38] T396073: Request to increase RAM and VCPU quota in codesearch - https://phabricator.wikimedia.org/T396073 [23:20:42] !log raymond-ndibe@cloudcumin1001 codesearch END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T396073) [23:21:48] !log raymond-ndibe@cloudcumin1001 codesearch START - Cookbook wmcs.openstack.quota_increase (T396073) [23:21:54] !log raymond-ndibe@cloudcumin1001 codesearch END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T396073) [23:24:15] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06collaboration-services: Request to increase RAM and VCPU quota in codesearch - https://phabricator.wikimedia.org/T396073#10886179 (10Raymond_Ndibe) **Before:** ` raymond-ndibe@cloudcontrol1006:~$ sudo wmcs-openstack quota show codesearch +---------------... [23:25:08] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06collaboration-services: Request to increase RAM and VCPU quota in codesearch - https://phabricator.wikimedia.org/T396073#10886180 (10Raymond_Ndibe) I'm guessing this solved this issue @Dzahn . If it didn't please reopen this or create another quota requ... [23:25:18] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06collaboration-services: Request to increase RAM and VCPU quota in codesearch - https://phabricator.wikimedia.org/T396073#10886181 (10Raymond_Ndibe) 05Open→03Resolved [23:51:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-10 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:56:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-10 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:56:34] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-10 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses