[00:04:07] (03open) 10ttaylor: Draft: Refactored project structure to add Python API to relay events [toolforge-repos/listen-to-wiki-changes] - 10https://gitlab.wikimedia.org/toolforge-repos/listen-to-wiki-changes/-/merge_requests/1 [00:06:21] FIRING: PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon [01:22:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [01:22:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [01:27:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [02:44:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [03:00:31] 10Tool-refill: Use full month name in date - https://phabricator.wikimedia.org/T393124#10794530 (10Ponor) I checked the references in [[London]] on a few major wikis. Some wikis use the ISO 8601 date format exclusively, some use their local date format or both. On all wikis the ISO date is transformed into the l... [03:13:31] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [03:34:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [03:54:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [04:02:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [04:12:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [04:59:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [04:59:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [05:07:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [05:09:18] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [05:09:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [05:24:18] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [05:42:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [06:31:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [06:46:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [06:48:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:58:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [08:03:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [08:41:53] (03PS9) 10Slyngshede: Add Bitu container [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318) [08:44:25] 10Toolforge (Toolforge iteration 19): [toolsbeta] update prometheus certs - https://phabricator.wikimedia.org/T393438 (10dcaro) 03NEW [08:44:38] 10Toolforge (Toolforge iteration 19): [toolsbeta] update prometheus certs - https://phabricator.wikimedia.org/T393438#10794841 (10dcaro) p:05Triage→03High a:03dcaro [08:44:42] 10Toolforge (Toolforge iteration 19): [toolsbeta] update prometheus certs - https://phabricator.wikimedia.org/T393438#10794844 (10dcaro) 05Open→03In progress [08:56:56] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [toolsbeta] update prometheus certs - https://phabricator.wikimedia.org/T393438#10794865 (10dcaro) [09:03:34] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [toolsbeta] update prometheus certs - https://phabricator.wikimedia.org/T393438#10794876 (10dcaro) Prometheus restarted, and it's starting to correctly pull data from all the endpoints: {F59718722} [09:04:37] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [toolsbeta] update prometheus certs - https://phabricator.wikimedia.org/T393438#10794879 (10dcaro) Finished, no unhealthy targets: {F59718739} [09:08:34] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [toolsbeta] update prometheus certs - https://phabricator.wikimedia.org/T393438#10794885 (10dcaro) 05In progress→03Resolved [09:14:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [09:18:15] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Modernize openstack rbac - https://phabricator.wikimedia.org/T330759#10794921 (10taavi) 05Stalled→03Open [09:24:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [09:42:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [09:42:59] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki nupwiki - https://phabricator.wikimedia.org/T390714#10795072 (10fnegri) 05In progress→03Resolved Re-ran the cookbook and the views were created successfully on `clouddb*` hosts. [09:45:54] (03PS1) 10Majavah: inventory: Use VXLAN/IPv6-dualstack as internal network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1142547 [09:48:11] (03approved) 10fnegri: Bug: T390056 importing tools volumes to Tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/19 (owner: 10chuckonwumelu) [10:07:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [10:27:21] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/100 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:27:26] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/100 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:27:31] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/69 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:27:37] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/69 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:27:39] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/28 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:27:43] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/28 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:28:29] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/239 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:28:31] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/239 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:30:18] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.103-20250506102749-ac0f961b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/756 [10:32:21] 10Tools, 03Wikimedia-Hackathon-2025: Start a rewrite the inat2wiki tool - https://phabricator.wikimedia.org/T392822#10795188 (10Aklapper) [10:37:37] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: [Session] Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10795215 (10Aklapper) @Sarai-WMF: Thanks for participating in the Hackathon! We hope you had a great time. * If this session too... [10:38:25] 10Tools, 03Wikimedia-Hackathon-2025: Velorail - https://phabricator.wikimedia.org/T393323#10795218 (10Aklapper) [10:39:10] 10Tools, 03Wikimedia-Hackathon-2025: Migrate Incubator Dashboard to public Superset + improvements - https://phabricator.wikimedia.org/T392813#10795223 (10Aklapper) [10:45:12] 06cloud-services-team, 10Cloud-VPS: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request - https://phabricator.wikimedia.org/T393024#10795253 (10taavi) a:03taavi [10:59:30] (03CR) 10Slyngshede: Add Bitu container (034 comments) [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318) (owner: 10Slyngshede) [11:00:32] (03PS10) 10Slyngshede: Add Bitu container [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318) [11:14:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:14:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:16:30] 06cloud-services-team, 10Cloud-VPS: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request - https://phabricator.wikimedia.org/T393024#10795303 (10taavi) One thing to note is that this is only happening with HTTP 2.0 requests, and seemingly only for requests that are not the fi... [11:19:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:20:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:24:18] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:25:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:28:52] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: [Session] Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10795347 (10dcaro) 05Open→03Resolved [11:35:44] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10795375 (10taavi) [11:51:46] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [11:54:17] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#10795507 (10dcaro) [12:04:48] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [12:05:39] (03approved) 10dcaro: components-api: bump to 0.0.103-20250506102749-ac0f961b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/756 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:05:42] (03merge) 10dcaro: components-api: bump to 0.0.103-20250506102749-ac0f961b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/756 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:05:52] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/63 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:05:55] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/63 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:06:20] 06cloud-services-team, 10Cloud-VPS: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request - https://phabricator.wikimedia.org/T393024#10795574 (10taavi) I tried to: * Handle possible Lua errors in the code, and set `lua_socket_log_errors off;` instead (as recommended in the lu... [12:08:21] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.66-20250506120606-dc233cef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/757 [12:08:23] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.66-20250506120606-dc233cef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/757 [12:09:40] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69 [12:11:18] 06cloud-services-team, 10Cloud-VPS: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request - https://phabricator.wikimedia.org/T393024#10795597 (10taavi) >>! In T393024#10795574, @taavi wrote: > Now I've set `loglevel verbose` in the Redis configuration in the hopes that that w... [12:12:58] 10Tool-kuwikibot, 10Toolhub: Invalid source code and issues URL on https://toolsadmin.wikimedia.org/tools/id/kuwikibot - https://phabricator.wikimedia.org/T361553#10795600 (10Aklapper) a:05Roj1512→03None Unassigning as account has no auth provider connected. Problem still exists though. [12:15:03] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69 [12:16:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [12:21:35] 06cloud-services-team, 10Cloud-VPS, 06Data-Engineering, 07IPv6, 13Patch-For-Review: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10795649 (10Gehel) Removing DPE SRE as it does not look like we need to be directly involved. Please add us back if needed! [12:28:04] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [12:28:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [12:28:56] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [12:30:35] (03update) 10dcaro: [jobs-cli] only send timeout if it's set by the user [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe) [12:33:38] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) (owner: 10raymond-ndibe) [12:34:07] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#10795682 (10dcaro) 05In progress→03Stalled [12:45:01] (03approved) 10dcaro: api-gateway: bump to 0.0.66-20250506120606-dc233cef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/757 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:45:05] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [12:52:31] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [12:55:26] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [13:01:48] (03update) 10dcaro: [jobs-cli] health_check and quota refactor [repos/cloud/toolforge/jobs-cli] (schedule_timeout_default_None) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe) [13:02:21] (03update) 10dcaro: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe) [13:02:44] (03update) 10dcaro: [jobs-cli] only send timeout if it's set by the user [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe) [13:03:52] (03update) 10dcaro: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] (move_most_custom_validations_out_of_api_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe) [13:04:50] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [13:05:21] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [13:06:08] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): [toolsdb] Use DNS CNAMEs instead of A records - https://phabricator.wikimedia.org/T392831#10795850 (10fnegri) 05Open→03In progress [13:08:30] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Service implementation for cloudcephosd2004-dev - https://phabricator.wikimedia.org/T392366#10795879 (10Andrew) 05Open→03Resolved [13:08:37] (03open) 10fnegri: toolsdb: use DNS CNAME instead of A records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/228 (https://phabricator.wikimedia.org/T392831) [13:08:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [13:12:40] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [13:15:25] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [13:21:30] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [13:42:26] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request - https://phabricator.wikimedia.org/T393024#10796079 (10taavi) The latest patch seems to have done the trick. I'm confident enough that I'm moving some tools-static... [13:59:18] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:07:00] 10Tools, 03Wikimedia-Hackathon-2025: Start a rewrite the inat2wiki tool - https://phabricator.wikimedia.org/T392822#10796194 (10TiagoLubiana) Some news on this: Dev version parallel to original, live at https://inat2wiki-dev.toolforge.org/ This development version includes support for translations and trans... [14:13:15] 06cloud-services-team, 10Cloud-VPS: gitlab ci: validate secrets settings in pipeline for tofu integration - https://phabricator.wikimedia.org/T391467#10796212 (10fnegri) 05Open→03Resolved a:03fnegri I found some GitLab documentation about this: > By default, pipelines from forked projects can’t acce... [14:21:27] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: gitlab ci: validate secrets settings in pipeline for tofu integration - https://phabricator.wikimedia.org/T391467#10796239 (10fnegri) [15:18:46] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [15:24:27] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [15:28:17] (03update) 10ttaylor: Draft: Refactored project structure to add Python API to relay events [toolforge-repos/listen-to-wiki-changes] - 10https://gitlab.wikimedia.org/toolforge-repos/listen-to-wiki-changes/-/merge_requests/1 [15:36:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:41:22] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:51:11] (03update) 10dcaro: crds: use the api name in the crd name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/68 (https://phabricator.wikimedia.org/T386829) [15:52:01] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [15:59:04] (03update) 10dcaro: [jobs-cli] only send timeout if it's set by the user [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/96 (owner: 10raymond-ndibe) [15:59:28] (03update) 10dcaro: [jobs-cli] only send timeout if it's set by the user [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe) [16:00:49] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [16:01:09] (03open) 10dcaro: cli: increase timeout to jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/101 [16:16:46] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Retire explicit 'roots' sudo policies - https://phabricator.wikimedia.org/T392797#10796817 (10fnegri) I support dropping the redundant list. I thought I filed a related phab task about simplifying root/admin lists, but I cannot find it so maybe I never... [16:17:22] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [16:21:39] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [16:25:05] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [16:35:30] (03update) 10dcaro: crds: use the api name in the crd name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/68 (https://phabricator.wikimedia.org/T386829) [16:36:50] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [16:38:39] (03update) 10dcaro: crds: use the api name in the crd name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/68 (https://phabricator.wikimedia.org/T386829) [16:39:16] (03update) 10dcaro: cli: increase timeout to jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/101 [16:44:21] (03merge) 10dcaro: api-gateway: bump to 0.0.66-20250506120606-dc233cef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/757 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:44:58] (03approved) 10andrew: cli: increase timeout to jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/101 (owner: 10dcaro) [16:47:18] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10796987 (10taavi) [16:51:32] (03merge) 10dcaro: cli: increase timeout to jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/101 [16:51:51] (03update) 10dcaro: [jobs-cli] only send timeout if it's set by the user [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe) [16:59:58] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/130 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:00:02] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/130 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:00:49] 10Tool-refill: Use full month name in date - https://phabricator.wikimedia.org/T393124#10797049 (10Ponor) Pull request: https://github.com/CurbSafeCharmer/refill/pull/264 [17:05:06] (03update) 10dcaro: crds: use the api name in the crd name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/68 (https://phabricator.wikimedia.org/T386829) [17:05:09] (03update) 10dcaro: crds: use the api name in the crd name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/68 (https://phabricator.wikimedia.org/T386829) [17:07:10] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.188-20250506170015-db3e79d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/758 [17:07:13] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.188-20250506170015-db3e79d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/758 [17:35:34] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496 (10dduvall) 03NEW [17:55:08] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797342 (10bd808) [17:57:00] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797345 (10bd808) [17:57:21] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797346 (10bd808) +1 [18:06:39] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797398 (10bd808) [18:38:11] !log andrew@cloudcumin1001 zuul3 START - Cookbook wmcs.openstack.quota_increase (T393496) [18:38:15] T393496: Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496 [18:38:20] !log andrew@cloudcumin1001 zuul3 END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T393496) [18:39:17] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797497 (10Andrew) 05Open→03Resolved a:03Andre... [18:45:06] !log andrew@cloudcumin1001 osmviews START - Cookbook wmcs.vps.create_project for project osmviews in eqiad1 (T393013) [18:45:07] andrew@cloudcumin1001: Unknown project "osmviews" [18:45:08] andrew@cloudcumin1001: Did you mean to say "tools.osmviews" instead? [18:45:08] T393013: Request creation of osmviews VPS project with 20 GiB object storage quota - https://phabricator.wikimedia.org/T393013 [18:45:41] (03open) 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49: projects: added project osmviews [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/229 (https://phabricator.wikimedia.org/T393013) [18:46:43] (03merge) 10andrew: projects: added project osmviews [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/229 (https://phabricator.wikimedia.org/T393013) (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [18:48:27] !log andrew@cloudcumin1001 osmviews END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project osmviews in eqiad1 (T393013) [18:48:27] andrew@cloudcumin1001: Unknown project "osmviews" [18:48:27] andrew@cloudcumin1001: Did you mean to say "tools.osmviews" instead? [18:49:21] 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of osmviews VPS project with 20 GiB object storage quota - https://phabricator.wikimedia.org/T393013#10797523 (10Andrew) 05Open→03Resolved a:03Andrew I've created the project with Sascha as the initial user. You can add additional u... [18:50:44] !log andrew@cloudcumin1001 misssearch START - Cookbook wmcs.vps.create_project for project misssearch in eqiad1 (T392903) [18:50:45] andrew@cloudcumin1001: Unknown project "misssearch" [18:50:46] T392903: Request creation of MissSearch VPS project - https://phabricator.wikimedia.org/T392903 [18:51:18] (03open) 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49: projects: added project misssearch [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/230 (https://phabricator.wikimedia.org/T392903) [18:52:27] (03merge) 10andrew: projects: added project misssearch [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/230 (https://phabricator.wikimedia.org/T392903) (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [18:57:58] 10Cloud-Services: Unable to create python virtualenv on Toolforge - https://phabricator.wikimedia.org/T393505 (10Huji) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific pro... [18:58:29] 06cloud-services-team, 10Toolforge: Unable to create python virtualenv on Toolforge - https://phabricator.wikimedia.org/T393505#10797555 (10Huji) [18:59:43] !log andrew@cloudcumin1001 misssearch END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project misssearch in eqiad1 (T392903) [18:59:46] T392903: Request creation of MissSearch VPS project - https://phabricator.wikimedia.org/T392903 [19:00:54] !log andrew@cloudcumin1001 misssearch START - Cookbook wmcs.openstack.quota_increase (T392903) [19:00:59] 06cloud-services-team, 10Toolforge: Unable to create python virtualenv on Toolforge - https://phabricator.wikimedia.org/T393505#10797564 (10taavi) 05Open→03Invalid Per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python virtual environments must be created within a job (or in the to-be-renamed `w... [19:01:01] !log andrew@cloudcumin1001 misssearch END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T392903) [19:01:46] 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of MissSearch VPS project - https://phabricator.wikimedia.org/T392903#10797570 (10Andrew) 05Open→03Resolved a:03Andrew You should be all set. Re-open this ticket if you run into trouble! [19:37:02] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10797690 (10Pastykros) Yeah, enabling IPv6 for your Cloud VPS web proxy on VikHost [[ https://vikhost.com/ | https://vikhost.com/ ]] is super straightforward.... [20:24:25] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Repurpose 5 config B servers - https://phabricator.wikimedia.org/T380805#10797841 (10Jclark-ctr) @Andrew can this be closed out if nothing else is needed? [20:31:44] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10797893 (10JeanFred) >>! In T294892#10779177, @Addshore wrote: >> Allow to pass the endpoint to pywikibot.data.spa... [20:34:42] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Modernize openstack rbac - https://phabricator.wikimedia.org/T330759#10797896 (10Andrew) [20:43:31] 06cloud-services-team: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10797917 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudrabbit2003-dev.codfw.wmnet with OS bookworm