[00:13:55] (03CR) 10CI reject: [V:04-1] build: Updating mediawiki/mediawiki-codesniffer to 48.0.0 [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1185449 (owner: 10Libraryupgrader) [05:01:12] 06cloud-services-team, 10Striker: Cannot connect developer account to Phabricator (Error updating database) - https://phabricator.wikimedia.org/T403918#11156057 (10JosefAnthony) yes is fixed now. thanks! [06:28:57] 06cloud-services-team, 10Toolforge: Dotnet bots failing with no logs - https://phabricator.wikimedia.org/T403927#11156096 (10taavi) [08:06:38] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [08:09:15] !log dcaro@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) [08:09:15] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11156350 (10dcaro) [08:09:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [08:09:20] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [08:09:49] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [08:11:52] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.29.15 to 1.30.14 (T402378) [08:11:57] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [08:13:31] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.29.15 to 1.30.14 (T402378) [08:14:34] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11156362 (10dcaro) [08:16:16] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.29.15 to 1.30.14 (T402378) [08:19:53] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [08:22:15] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [08:23:31] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [08:26:56] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.29.15 to 1.30.14 (T402378) [08:27:01] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [08:27:15] RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [08:28:35] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.29.15 to 1.30.14 (T402378) [08:28:41] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11156409 (10dcaro) Maybe create a small set of tests for running during upgrades instead of the whole of it. [08:32:02] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [08:32:49] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [08:34:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [08:35:17] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [08:37:37] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.29.15 to 1.30.14 (T402378) [08:37:42] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [08:37:43] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.29.15 to 1.30.14 (T402378) [08:38:08] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [08:39:14] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [08:40:35] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [08:40:36] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [08:42:15] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-9.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [08:46:28] dcaro@cloudcumin1001 upgrade (PID 1686722) is awaiting input [08:47:03] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.29.15 to 1.30.14 (T402378) [08:47:08] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [08:47:16] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [08:49:12] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11156568 (10dcaro) [08:52:29] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 (T402378) [08:52:34] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [08:58:43] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 (T402378) [08:58:48] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [08:59:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [08:59:22] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [09:10:26] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 (T402378) [09:10:31] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [09:17:29] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [09:17:35] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [09:18:12] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 (T402378) [09:18:15] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [09:20:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [09:20:18] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [09:20:18] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [09:20:18] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:22:12] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [09:22:12] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [09:22:13] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [09:22:13] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:22:51] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [09:22:51] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [09:22:51] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [09:22:51] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:32:03] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [09:32:04] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [09:32:04] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [09:32:04] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:32:07] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [09:32:54] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [09:32:54] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [09:32:54] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [09:32:54] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:37:09] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [09:38:42] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [09:38:42] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [09:38:42] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [09:38:42] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:38:44] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [09:40:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [09:40:00] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [09:40:00] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [09:40:00] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:40:17] !log dcaro@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=97) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w [09:40:17] orker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wo [09:40:17] rker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-wor [09:40:17] ker-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:40:42] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [09:40:43] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [09:40:43] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [09:40:43] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:40:56] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [09:42:09] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [09:42:09] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [09:42:10] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [09:42:10] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:43:40] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor [09:43:40] ker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-work [09:43:40] er-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worke [09:43:40] r-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:44:07] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [09:44:08] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [09:44:08] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [09:44:08] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:44:10] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [09:46:29] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [09:46:29] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [09:46:29] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [09:46:29] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:49:24] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [09:49:24] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [09:49:24] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [09:49:24] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:49:28] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [09:50:01] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [09:50:01] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [09:50:01] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [09:50:01] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:55:20] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [09:55:20] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [09:55:20] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [09:55:20] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [09:55:24] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [09:58:54] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68: (T402378) [09:58:55] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-68: (T402378) [09:58:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:58:58] 10VPS-project-Phabricator, 06collaboration-services: 'Fulltext' searches fail on the test Phabricator instance (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948 (10A_smart_kitten) 03NEW [09:59:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:59:00] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 (T402378) [09:59:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:00:50] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11156839 (10fnegri) a:03fnegri [10:00:56] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [10:01:04] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [10:05:50] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 (T402378) [10:05:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:05:55] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [10:06:15] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [10:06:15] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [10:06:16] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [10:06:16] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [10:06:37] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [10:06:37] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [10:06:37] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [10:06:37] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [10:06:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [10:06:53] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [10:06:53] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [10:06:53] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [10:07:13] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team: 'Fulltext' searches fail on the test Phabricator instance (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11156855 (10LSobanski) [10:09:53] dcaro@cloudcumin1001 upgrade_workers (PID 1699715) is awaiting input [10:16:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [10:16:00] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [10:16:00] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [10:16:00] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [10:16:04] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [10:20:32] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [10:26:08] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor [10:26:08] ker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-work [10:26:08] er-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worke [10:26:08] r-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [10:26:12] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [10:26:58] (03merge) 10mhorsey: Close button editing feature [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/6 (owner: 10vriaa) [10:27:14] (03merge) 10mhorsey: feat: Add banner code generation feature [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/8 (owner: 10vriaa) [10:28:49] (03update) 10mhorsey: fix: Move Close button out of banner link [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/9 (owner: 10vriaa) [10:29:07] (03merge) 10mhorsey: fix: Move Close button out of banner link [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/9 (owner: 10vriaa) [10:29:29] (03update) 10mhorsey: fix: Reactivity issues in useUpdateSide composable [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/10 (owner: 10vriaa) [10:29:35] (03merge) 10mhorsey: fix: Reactivity issues in useUpdateSide composable [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/10 (owner: 10vriaa) [10:29:50] (03update) 10mhorsey: fix: BorderProperty toggle state not syncing with parent updates [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/11 (owner: 10vriaa) [10:29:59] (03merge) 10mhorsey: fix: BorderProperty toggle state not syncing with parent updates [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/11 (owner: 10vriaa) [10:30:12] (03update) 10mhorsey: fix: MarginProperty and PaddingProperty not syncing with parent state [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/12 (owner: 10vriaa) [10:30:19] (03merge) 10mhorsey: fix: MarginProperty and PaddingProperty not syncing with parent state [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/12 (owner: 10vriaa) [10:30:33] (03update) 10mhorsey: fix: Sync toggle with colorModel changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/13 (owner: 10vriaa) [10:30:42] (03merge) 10mhorsey: fix: Sync toggle with colorModel changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/13 (owner: 10vriaa) [10:30:56] (03update) 10mhorsey: fix: ColorPicker component ignoring modelValue updates [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/14 (owner: 10vriaa) [10:31:02] (03merge) 10mhorsey: fix: ColorPicker component ignoring modelValue updates [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/14 (owner: 10vriaa) [10:31:14] (03update) 10mhorsey: fix: BasicDropdown not updating when modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/15 (owner: 10vriaa) [10:31:19] (03merge) 10mhorsey: fix: BasicDropdown not updating when modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/15 (owner: 10vriaa) [10:31:34] (03update) 10mhorsey: fix: AddLink not syncing with parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/16 (owner: 10vriaa) [10:31:37] (03merge) 10mhorsey: fix: AddLink not syncing with parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/16 (owner: 10vriaa) [10:31:50] (03update) 10mhorsey: fix: useSelectIconButtonOption not responding to parent changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/17 (owner: 10vriaa) [10:31:56] (03merge) 10mhorsey: fix: useSelectIconButtonOption not responding to parent changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/17 (owner: 10vriaa) [10:32:13] (03update) 10mhorsey: fix: TextTransform dropdown not responding to parent changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/18 (owner: 10vriaa) [10:32:21] (03merge) 10mhorsey: fix: TextTransform dropdown not responding to parent changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/18 (owner: 10vriaa) [10:32:34] (03update) 10mhorsey: fix: BannerLink not responding to parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/19 (owner: 10vriaa) [10:33:29] (03merge) 10mhorsey: fix: BannerLink not responding to parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/19 (owner: 10vriaa) [10:33:52] (03update) 10mhorsey: fix: Image components not responding to parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/20 (owner: 10vriaa) [10:34:02] (03merge) 10mhorsey: fix: Image components not responding to parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/20 (owner: 10vriaa) [10:35:28] 10VPS-project-Phabricator, 06collaboration-services, 10Release-Engineering-Team (Doing 😎): 'Fulltext' searches fail on the test Phabricator instance (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11156958 (10Aklapper) p:05Tria... [10:36:36] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor [10:36:36] ker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-work [10:36:36] er-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worke [10:36:36] r-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-9 (T402378) [10:36:39] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [10:46:10] 10VPS-project-Phabricator, 06collaboration-services, 13Patch-For-Review, 10Release-Engineering-Team (Doing 😎): 'Fulltext' searches fail on the test Phabricator instance (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11156998 (... [10:50:00] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T403348#11157006 (10fnegri) 05Open→03Resolved [10:53:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-68 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [10:56:09] 06cloud-services-team, 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401076#11157016 (10github-toolforge-bot) dhinus opened https://github.com/toolforge/paws/pull/497 [10:56:34] dhinus opened https://github.com/toolforge/paws/pull/497 [11:06:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [11:07:26] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11157038 (10dcaro) [11:08:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 (T402378) [11:08:14] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [11:12:45] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 (T402378) [11:13:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud (T402378) [11:13:57] T402378: [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 [11:14:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud (T402378) [11:14:51] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11157053 (10dcaro) [11:15:31] (03CR) 10David Caro: [C:03+2] k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [11:15:34] (03CR) 10David Caro: [C:03+2] k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [11:15:37] (03CR) 10David Caro: [C:03+2] k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [11:15:39] (03CR) 10David Caro: [C:03+2] k8s.worker.upgrade: allow not passing src_version [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183623 (owner: 10David Caro) [11:15:42] (03CR) 10David Caro: [C:03+2] k8s.prepare_upgrade: remove not used src-version param [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183620 (owner: 10David Caro) [11:19:49] (03Merged) 10jenkins-bot: k8s.prepare_upgrade: remove not used src-version param [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183620 (owner: 10David Caro) [11:19:49] (03Merged) 10jenkins-bot: k8s.worker.upgrade: allow not passing src_version [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183623 (owner: 10David Caro) [11:20:49] (03Merged) 10jenkins-bot: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [11:20:51] (03Merged) 10jenkins-bot: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [11:20:53] (03Merged) 10jenkins-bot: k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [11:29:31] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [11:29:38] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [11:29:58] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-68 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [11:31:54] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11157087 (10dcaro) [11:46:39] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [11:46:44] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [11:55:31] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11157113 (10dcaro) >>! In T402378#11156717, @dcaro wrote: > Just noticed that the containers restarts graph in toolsbeta is empty, and in tools i... [11:55:44] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11157116 (10dcaro) 05In progress→03Resolved [11:56:08] (03merge) 10dcaro: k8s: upgrade to 1.30 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/241 (https://phabricator.wikimedia.org/T362869) [11:56:29] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#11157119 (10dcaro) [12:07:14] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [12:07:20] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [12:10:17] 06cloud-services-team, 10Toolforge: [jobs-cli] jobs are being updated (deleted/created) when no changes present - https://phabricator.wikimedia.org/T403760#11157152 (10DamianZaremba) Looks like this is working as expected now ` tools.cluebotng-review@tools-bastion-13:~$ toolforge jobs load jobs.yaml INFO: lo... [12:16:16] !log dcaro@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) [12:17:21] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [jobs-cli] jobs are being updated (deleted/created) when no changes present - https://phabricator.wikimedia.org/T403760#11157157 (10dcaro) a:03Raymond_Ndibe [12:17:52] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [jobs-cli] jobs are being updated (deleted/created) when no changes present - https://phabricator.wikimedia.org/T403760#11157160 (10dcaro) 05Open→03Resolved Awesome :) [12:18:39] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [jobs-api] Allow customizing time to request Loki logs for - https://phabricator.wikimedia.org/T400917#11157162 (10dcaro) [12:19:51] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1185912 (owner: 10L10n-bot) [12:25:03] (03open) 10dcaro: package: upgrade all deps [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/64 [12:33:53] (03update) 10dcaro: package: upgrade all deps [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/64 [12:40:38] 06cloud-services-team, 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401076#11157261 (10github-toolforge-bot) dhinus closed https://github.com/toolforge/paws/pull/497 [12:41:51] dhinus closed https://github.com/toolforge/paws/pull/497 [13:04:25] (03open) 10dcaro: pacakage: bump dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/141 [13:04:56] (03update) 10dcaro: package: upgrade all deps [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/64 [13:05:57] (03PS1) 10Muehlenhoff: Add dummy keytab for install3004 [labs/private] - 10https://gerrit.wikimedia.org/r/1185924 (https://phabricator.wikimedia.org/T402259) [13:06:41] 06cloud-services-team, 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962 (10DamianZaremba) 03NEW [13:09:24] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Add dummy keytab for install3004 [labs/private] - 10https://gerrit.wikimedia.org/r/1185924 (https://phabricator.wikimedia.org/T402259) (owner: 10Muehlenhoff) [13:09:52] 06cloud-services-team, 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401076#11157390 (10fnegri) 05Open→03Resolved PAWS upgraded, I verified that new PAWS servers are getting pywikibot 10.4.0. [13:19:08] (03open) 10vriaa: fix: Broken main [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/22 [13:26:05] 10Toolforge (Quota-requests): Request increased quota for cluebotng-review Toolforge tool - https://phabricator.wikimedia.org/T403964 (10DamianZaremba) 03NEW [13:29:47] (03update) 10dcaro: pacakage: bump dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/141 [13:35:34] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157547 (10fnegri) The disk is full: ` /dev/sdb 20G 19G 0 100% /var/lib/mysql ` Mariadb logs contain messsages like: ` 2025-09-08 2:16:38 92654 [ERROR] InnoDB: preallocating 15309209... [13:36:44] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157550 (10fnegri) Most of the space is taken by `mariadb-general.log` ` root@cluebotng-review:/var/lib/mysql/data# du -hs * 32K aria_log.00000001 4.0K aria_log_control 2.9G cbng_reviewer 4... [13:38:45] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157555 (10DamianZaremba) > Most of the space is taken by mariadb-general.log I tried `Discard log` from the UI yesterday in case this was the issue, but perhaps that only targets the file in the obj... [13:40:25] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157571 (10DamianZaremba) > I tried Discard log from the UI yesterday in case this was the issue, but perhaps that only targets the file in the object storage. The usage is shown on the dashbaord: {F6... [13:42:48] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157585 (10DamianZaremba) > Mariadb logs contain messsages like: Those seem expected after the disk is full (innodb cannot write temp files). Perhaps there is something more useful in the head which... [13:45:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_mons [13:47:18] (03update) 10dcaro: Expand reuse_from validation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/126 (https://phabricator.wikimedia.org/T403287) (owner: 10damian) [13:48:04] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157606 (10fnegri) > Perhaps there is something more useful in the head which explains why the file got so large, perhaps slow queries? I think it is logging all queries, not just slow ones. It doesn... [13:48:12] 10Toolforge (Quota-requests): Request increased quota for cluebotng-review Toolforge tool - https://phabricator.wikimedia.org/T403964#11157608 (10DamianZaremba) With the default memory/cpu: ` tools.cluebotng-review@tools-bastion-13:~$ toolforge jobs list +-------------------------------------------+-------------... [13:50:18] PROBLEM - Host cloudcephmon1004 is DOWN: PING CRITICAL - Packet loss = 100% [13:51:48] RECOVERY - Host cloudcephmon1004 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [13:52:04] 06cloud-services-team, 10Cloud-VPS, 07Security: Move cloud-wide root keys to the main puppet repo - https://phabricator.wikimedia.org/T317362#11157621 (10fgiunchedi) [13:52:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:52:12] 06cloud-services-team, 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962#11157622 (10DamianZaremba) > Currently jobs-api forces limit and request to the same value, if less than 0.5Gi memory or 500m cpu is configured Actually this also applies to the default va... [13:54:18] (03merge) 10mhorsey: fix: Broken main [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/22 (owner: 10vriaa) [13:56:02] PROBLEM - Host cloudcephmon1005 is DOWN: PING CRITICAL - Packet loss = 100% [13:57:27] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157633 (10fnegri) File truncated, but it's already growing again. `general_log` is set to 1: ` MariaDB [(none)]> SELECT @@general_log; +---------------+ | @@general_log | +---------------+ |... [13:57:32] RECOVERY - Host cloudcephmon1005 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [13:57:44] 06cloud-services-team, 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962#11157635 (10DamianZaremba) With default requests (`0.5`): ` tools.cluebotng-review@tools-bastion-13:~$ kubectl describe quota Name: tool-cluebotng-review Namespace:... [13:59:13] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157658 (10fnegri) Can you try clicking `Disable log` now? {F65986128} [13:59:56] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157668 (10DamianZaremba) It might be from me 'experimenting' with trying to get the logs from the dashboard (they are currently reporting as `Disabled` though). There are quite a lot of queries run... [14:00:53] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157680 (10fnegri) Yeah you're right it says "disabled", maybe try enabling and disabling again? There is something that is not quite working. [14:01:32] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157692 (10DamianZaremba) > Can you try clicking Disable log now? {F65986135} {F65986137} [14:02:20] PROBLEM - Host cloudcephmon1006 is DOWN: PING CRITICAL - Packet loss = 100% [14:03:48] RECOVERY - Host cloudcephmon1006 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [14:05:09] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.upgrade_mons (exit_code=0) [14:05:23] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157710 (10fnegri) Hmpf, that did not update the config file, or the database setting. [14:07:23] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157728 (10DamianZaremba) I also tried enable, saw it reported as ready, then disable FYI. It did take 'a while' when clicking the button, but apparently didn't do much. [14:08:20] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157733 (10fnegri) I tried again "Enable" then "Disable" and that seems to have fixed it, the config file was deleted: ` root@cluebotng-review:/var/lib/mysql/data# cat /etc/mysql/conf.d/10-system-003... [14:10:51] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157739 (10fnegri) Tried restarting the database as well, it's still disabled. We should be good for now. ` MariaDB [(none)]> SELECT @@general_log; +---------------+ | @@general_log | +--------------... [14:12:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:14:48] 06cloud-services-team, 10Toolforge: Root access to Trove instance - https://phabricator.wikimedia.org/T403920#11157748 (10DamianZaremba) Great, I'll try running one of the import jobs again in a few [14:21:03] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: [trove] disk full for cluebotng-review instance - https://phabricator.wikimedia.org/T403920#11157794 (10fnegri) 05Open→03In progress p:05Triage→03High [14:24:03] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: [trove] disk full for cluebotng-review instance - https://phabricator.wikimedia.org/T403920#11157810 (10DamianZaremba) Looks like things are processing as they should. It would be good to get a solution for the root access both so I can deploy https://github.... [14:28:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:31:25] (03update) 10dcaro: [toolforge_deploy_mr.py] support deploy of MRs from external contributors [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/273 (https://phabricator.wikimedia.org/T394595) (owner: 10raymond-ndibe) [14:38:21] (03open) 10dcaro: package: upgrade deps [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/35 [14:40:55] (03open) 10vriaa: fix: Capitalize CloseIcon file name correctly [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/23 [14:59:04] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11158045 (10Tgr) a:03matmarex [14:59:44] (03open) 10dcaro: package: upgrade dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/29 [15:05:27] 06cloud-services-team, 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962#11158103 (10DamianZaremba) This behaviour changed from 'less than the default' to 'less than or equal to the default' in https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_... [15:07:10] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [15:07:30] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: [trove] disk full for cluebotng-review instance - https://phabricator.wikimedia.org/T403920#11158116 (10fnegri) Yes let's create a separate ticket, and resolve this one is the urgent issue is fixed. [15:08:29] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [15:11:58] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [15:13:09] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [15:15:16] (03open) 10vriaa: fix(templates): ensure properties conform to expected schema to prevent runtime/type errors [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/24 [15:16:37] (03approved) 10dcaro: envvars-api: bump to 0.0.74-20250904080401-f39b2dc6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/954 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:16:40] (03update) 10dcaro: envvars-api: bump to 0.0.74-20250904080401-f39b2dc6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/954 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:16:53] (03update) 10dcaro: image-config: bump to 0.0.22-20250904080448-f3620fc9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/953 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:16:55] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component image-config [15:17:46] (03merge) 10dcaro: envvars-api: bump to 0.0.74-20250904080401-f39b2dc6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/954 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:18:06] (03open) 10dcaro: package: upgrade dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/25 [15:18:30] (03update) 10dcaro: image-config: bump to 0.0.22-20250904080448-f3620fc9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/953 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:27:33] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config [15:30:00] (03update) 10dcaro: package: upgrade dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/29 [15:30:09] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Solution for trove instance access without vps instance - https://phabricator.wikimedia.org/T403977 (10DamianZaremba) 03NEW [15:30:14] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: [trove] disk full for cluebotng-review instance - https://phabricator.wikimedia.org/T403920#11158332 (10DamianZaremba) 05In progress→03Resolved [15:30:25] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: [trove] disk full for cluebotng-review instance - https://phabricator.wikimedia.org/T403920#11158334 (10DamianZaremba) > Yes let's create a separate ticket, and resolve this one is the urgent issue is fixed. Done as T403977 [15:31:29] (03update) 10dcaro: package: upgrade dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/25 [15:32:02] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component image-config [15:36:14] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Solution for trove instance access without vps instance - https://phabricator.wikimedia.org/T403977#11158383 (10fnegri) There are two types of root access: 1 .SQL root access, as described in https://wikitech.wikimedia.org/wiki/Help:Trove_database_user_guide#... [15:38:28] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Solution for trove instance access without vps instance - https://phabricator.wikimedia.org/T403977#11158413 (10fnegri) p:05Triage→03Medium [15:41:25] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config [15:44:02] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Solution for trove instance access without vps instance - https://phabricator.wikimedia.org/T403977#11158438 (10DamianZaremba) 1. can be what this ticket covers 2. is complicated by having no monitoring/metrics or real insight beyond what horizon (mostly doesn... [15:51:37] (03approved) 10dcaro: image-config: bump to 0.0.22-20250904080448-f3620fc9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/953 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:51:41] (03merge) 10dcaro: image-config: bump to 0.0.22-20250904080448-f3620fc9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/953 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:20:00] 06cloud-services-team, 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962#11158687 (10dcaro) Trying to understand the issue, let me try to summarize, as it seems there's more than one thing you are reporting: * It's not possible to specify less than 100Mi/50m of... [16:30:32] 06cloud-services-team, 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962#11158763 (10dcaro) still using a lower request than limit might be useful to be able to allocate jobs, though lowering the default request might also be a good idea (and maybe an alternative). [16:30:48] (03open) 10dcaro: quota: use the same value for `request` and `limit` [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/74 [16:35:22] 10Toolforge (Quota-requests): Request increased quota for cluebotng-review Toolforge tool - https://phabricator.wikimedia.org/T403964#11158815 (10dcaro) +1 this should help too with the memory/cpu though https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/74 [16:36:00] 10Toolforge (Quota-requests): Request increased quota for cluebotng-review Toolforge tool - https://phabricator.wikimedia.org/T403964#11158819 (10dcaro) a:03fnegri [16:42:55] (03update) 10dcaro: quota: use the same value for `request` and `limit` [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/74 [16:49:21] (03update) 10dcaro: package: upgrade deps [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/35 [16:50:46] (03update) 10dcaro: package: upgrade dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/25 [16:51:17] (03update) 10dcaro: package: upgrade deps [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/35 [16:52:04] (03update) 10dcaro: package: upgrade dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/29 [17:06:47] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [17:17:31] 10VPS-project-voterlists, 06Trust-and-Safety, 03Elections: Create voter list for Wikimedia France board of administrators election - https://phabricator.wikimedia.org/T403809#11158991 (10SD0001) @asirac I didn't notice this earlier, but the criteria you've given refers to dates in the future: ` The sum of e... [17:38:05] 06cloud-services-team, 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962#11159106 (10DamianZaremba) > Trying to understand the issue, let me try to summarize, as it seems there's more than one thing you are reporting: Sorry, this was a bit of a process of discov... [17:49:28] 10Toolforge (Quota-requests): Request increased quota for cluebotng-review Toolforge tool - https://phabricator.wikimedia.org/T403964#11159150 (10DamianZaremba) > +1 this should help too with the memory/cpu though https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/74 This will... [19:02:49] FIRING: ObjectStorageSizeQuotaFull: Object storage quota by 'size' is 80.84% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageSizeQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageSizeQuotaFull [19:02:51] PROBLEM - Host cloudcephmon1004 is DOWN: PING CRITICAL - Packet loss = 100% [19:04:19] RECOVERY - Host cloudcephmon1004 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [19:24:27] (03update) 10vriaa: feat: Make editor responsive [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/21 [19:51:01] 06cloud-services-team, 10Horizon: Horizon web proxy creation "Forbidden (HTTP 403)" error - https://phabricator.wikimedia.org/T404002 (10Mhurd) 03NEW [19:53:32] (03open) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [19:58:28] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:11:53] 06cloud-services-team, 10Horizon: Horizon web proxy creation "Forbidden (HTTP 403)" error - https://phabricator.wikimedia.org/T404002#11159904 (10Mhurd) Will try again later today/tomorrow [20:16:17] (03update) 10raymond-ndibe: [helm_publish, image_publish]: skip harbor helm and image publish if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:17:04] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:17:20] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:17:31] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:18:27] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:29:55] 06cloud-services-team, 10Horizon: Horizon web proxy creation "Forbidden (HTTP 403)" error - https://phabricator.wikimedia.org/T404002#11159962 (10bd808) https://openstack-browser.toolforge.org/proxy/ tells me that the [[https://openstack-browser.toolforge.org/project/reading-web-staging|reading-web-staging]] p... [20:30:55] 06cloud-services-team, 10Horizon: Horizon web proxy creation "Forbidden (HTTP 403)" error - https://phabricator.wikimedia.org/T404002#11159967 (10Mhurd) @bd808 Ah thank you good to know!! Looking... [20:33:28] 06cloud-services-team, 10Horizon: Horizon web proxy creation "Forbidden (HTTP 403)" error - https://phabricator.wikimedia.org/T404002#11160004 (10bd808) "Can't edit backend of another project" in the error message was the clue that the name was taken. This error message being more directly something like "Back... [20:34:13] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [20:44:47] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [20:46:08] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [21:06:18] 06cloud-services-team, 10Horizon: Horizon web proxy creation "Forbidden (HTTP 403)" error - https://phabricator.wikimedia.org/T404002#11160103 (10Mhurd) Makes sense thanks!! [21:18:49] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [21:20:10] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [21:29:27] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [22:06:43] 06cloud-services-team, 10Horizon: Horizon web proxy creation "Forbidden (HTTP 403)" error - https://phabricator.wikimedia.org/T404002#11160422 (10Mhurd) 05Open→03Invalid [22:59:15] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [22:59:52] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [23:03:04] FIRING: ObjectStorageSizeQuotaFull: Object storage quota by 'size' is 80.84% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageSizeQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageSizeQuotaFull