[00:07:17] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:12:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:12:47] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:17:32] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:23:07] 06cloud-services-team, 10Cloud-VPS: wmcs Trixie kernel reboots - https://phabricator.wikimedia.org/T410846#11403652 (10Andrew) [01:14:22] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T408783) [01:14:25] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T408783) [01:14:26] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T408783) [01:14:42] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/84 (https://phabricator.wikimedia.org/T408783) [01:14:43] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/84 (https://phabricator.wikimedia.org/T408783) [01:14:48] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/84 (https://phabricator.wikimedia.org/T408783) [01:14:56] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/140 (https://phabricator.wikimedia.org/T408783) [01:14:57] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/140 (https://phabricator.wikimedia.org/T408783) [01:15:01] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/140 (https://phabricator.wikimedia.org/T408783) [01:15:06] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/58 (https://phabricator.wikimedia.org/T408783) [01:15:09] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/58 (https://phabricator.wikimedia.org/T408783) [01:15:12] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/58 (https://phabricator.wikimedia.org/T408783) [01:15:18] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T408783) [01:15:20] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T408783) [01:15:23] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T408783) [01:15:29] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/125 (https://phabricator.wikimedia.org/T408783) [01:15:30] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/125 (https://phabricator.wikimedia.org/T408783) [01:15:33] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/125 (https://phabricator.wikimedia.org/T408783) [01:16:05] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/101 (https://phabricator.wikimedia.org/T408783) [01:16:06] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/101 (https://phabricator.wikimedia.org/T408783) [01:16:10] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/101 (https://phabricator.wikimedia.org/T408783) [01:16:14] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/68 (https://phabricator.wikimedia.org/T408783) [01:16:17] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/68 (https://phabricator.wikimedia.org/T408783) [01:16:19] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/68 (https://phabricator.wikimedia.org/T408783) [01:16:43] (03update) 10raymond-ndibe: [dev] CONTRIBUTING.md [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/297 (https://phabricator.wikimedia.org/T408783) [01:16:46] (03approved) 10raymond-ndibe: [dev] CONTRIBUTING.md [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/297 (https://phabricator.wikimedia.org/T408783) [01:16:48] (03update) 10raymond-ndibe: [dev] CONTRIBUTING.md [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/297 (https://phabricator.wikimedia.org/T408783) [01:17:50] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.85-20251125011456-f16673c1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1088 (https://phabricator.wikimedia.org/T408783) [01:18:05] (03merge) 10raymond-ndibe: [dev] CONTRIBUTING.md [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/297 (https://phabricator.wikimedia.org/T408783) [01:18:17] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/68 (https://phabricator.wikimedia.org/T408783) [01:18:43] (03approved) 10raymond-ndibe: disable_tool.py: fix mypy error [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/25 [01:18:52] (03merge) 10raymond-ndibe: disable_tool.py: fix mypy error [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/25 [01:19:45] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T408783) [01:21:47] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: logs-api: bump to 0.0.8-20251125011956-f50aaf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1089 (https://phabricator.wikimedia.org/T408783) [01:21:55] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/33 (https://phabricator.wikimedia.org/T408783) [01:21:57] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/33 (https://phabricator.wikimedia.org/T408783) [01:22:35] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/33 (https://phabricator.wikimedia.org/T408783) [01:22:44] (03update) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/80 (https://phabricator.wikimedia.org/T408783) [01:22:48] (03approved) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/80 (https://phabricator.wikimedia.org/T408783) [01:22:55] (03merge) 10raymond-ndibe: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/80 (https://phabricator.wikimedia.org/T408783) [01:25:28] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.188-20251125012315-d226fa18 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1090 (https://phabricator.wikimedia.org/T408783) [01:26:25] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.69-20251125012254-2c085c20 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1091 (https://phabricator.wikimedia.org/T408783) [01:27:50] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [01:36:09] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [01:36:10] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [01:45:35] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [01:45:36] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [01:46:56] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11403854 (10Raymond_Ndibe) 05Open→03In progress [01:49:51] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [01:49:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [01:53:54] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [01:53:55] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [02:10:51] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [02:10:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [02:29:08] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [02:29:09] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [02:38:17] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [02:38:18] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [02:47:56] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [03:17:06] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.69-20251125012254-2c085c20 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1091 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:17:08] (03approved) 10raymond-ndibe: registry-admission: bump to 0.0.69-20251125012254-2c085c20 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1091 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:17:11] (03merge) 10raymond-ndibe: registry-admission: bump to 0.0.69-20251125012254-2c085c20 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1091 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:17:19] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.188-20251125012315-d226fa18 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1090 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:17:21] (03approved) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.188-20251125012315-d226fa18 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1090 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:17:23] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.188-20251125012315-d226fa18 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1090 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:17:46] (03merge) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.188-20251125012315-d226fa18 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1090 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:21:49] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.85-20251125011456-f16673c1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1088 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:21:49] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.85-20251125011456-f16673c1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1088 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:22:28] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.85-20251125011456-f16673c1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1088 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:22:53] (03update) 10raymond-ndibe: logs-api: bump to 0.0.8-20251125011956-f50aaf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1089 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:22:57] (03approved) 10raymond-ndibe: logs-api: bump to 0.0.8-20251125011956-f50aaf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1089 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:23:03] (03update) 10raymond-ndibe: logs-api: bump to 0.0.8-20251125011956-f50aaf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1089 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [03:23:29] (03merge) 10raymond-ndibe: logs-api: bump to 0.0.8-20251125011956-f50aaf4b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1089 (https://phabricator.wikimedia.org/T408783) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [04:05:47] (03update) 10raymond-ndibe: images: resolve the image every time [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 (owner: 10dcaro) [04:08:49] (03update) 10raymond-ndibe: images: resolve the image every time [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 (owner: 10dcaro) [04:43:06] (03update) 10raymond-ndibe: images: resolve the image every time [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 (owner: 10dcaro) [04:43:12] (03approved) 10raymond-ndibe: images: resolve the image every time [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 (owner: 10dcaro) [04:59:08] (03update) 10raymond-ndibe: images: support harbor-based pre-built images [repos/cloud/toolforge/jobs-api] (resolve_harbor_images_every_time) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/252 (https://phabricator.wikimedia.org/T409727) (owner: 10dcaro) [04:59:31] (03update) 10raymond-ndibe: images: support harbor-based pre-built images [repos/cloud/toolforge/jobs-api] (resolve_harbor_images_every_time) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/252 (https://phabricator.wikimedia.org/T409727) (owner: 10dcaro) [05:51:47] (03unapproved) 10raymond-ndibe: images: resolve the image every time [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 (owner: 10dcaro) [05:52:31] (03update) 10raymond-ndibe: images: resolve the image every time [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 (owner: 10dcaro) [08:12:37] 06cloud-services-team, 10Cloud-VPS: VM metadata service slow response - https://phabricator.wikimedia.org/T410983 (10fgiunchedi) 03NEW [08:19:24] 06cloud-services-team, 10Cloud-VPS: VM metadata service slow response - https://phabricator.wikimedia.org/T410983#11404183 (10fgiunchedi) [08:23:09] 06cloud-services-team, 10Cloud-VPS: VM metadata service slow response - https://phabricator.wikimedia.org/T410983#11404185 (10fgiunchedi) [08:29:07] 06cloud-services-team, 10Cloud-VPS: VM metadata service slow response - https://phabricator.wikimedia.org/T410983#11404216 (10fgiunchedi) [08:54:05] 06cloud-services-team, 10Cloud-VPS: VM metadata service slow response - https://phabricator.wikimedia.org/T410983#11404276 (10fgiunchedi) p:05Triage→03Low [08:56:13] (03CR) 10Filippo Giunchedi: [C:03+1] labs: add infra-tracing-nfs account [labs/private] - 10https://gerrit.wikimedia.org/r/1210591 (https://phabricator.wikimedia.org/T399313) (owner: 10Volans) [09:09:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:14:19] 10Tool-yearinreview, 10MediaWiki-extensions-Translate, 06Wikipedia-Android-App-Backlog, 06LPL Essential (FY26 Q2), and 3 others: PLURAL syntax validator gets confused by other uses of equals signs in the message, as seen at [[Wikimedia:Wikipedia-android-s... - https://phabricator.wikimedia.org/T409655#11404383 [09:37:14] 10Tool-erinnermich: [ErinnerMich] Add a blocklist for sites where requests should be rejected - https://phabricator.wikimedia.org/T410993 (10Tkarcher) 03NEW [09:37:54] 06cloud-services-team, 10Toolforge, 06Wikimedia Enterprise: Toolforge no longer has IP-based access to Wikimedia Enterprise - https://phabricator.wikimedia.org/T410994 (10Urbanecm) 03NEW [09:43:28] 10Tool-erinnermich: [ErinnerMich] Add a blocklist for sites where requests should be rejected - https://phabricator.wikimedia.org/T410993#11404527 (10Tkarcher) Pinging @M-J (to keep you in the loop on the planned changes) [10:19:53] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.instance.stop_start vm ntp-5 (cluster eqiad1) [10:20:21] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm ntp-5 (cluster eqiad1) [10:21:06] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-prometheus-2 (cluster eqiad1) [10:21:44] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-prometheus-2 (cluster eqiad1) [10:59:57] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11404870 (10taavi) [11:49:18] (03CR) 10Volans: [C:03+2] labs: add infra-tracing-nfs account [labs/private] - 10https://gerrit.wikimedia.org/r/1210591 (https://phabricator.wikimedia.org/T399313) (owner: 10Volans) [11:49:27] (03CR) 10Volans: [V:03+2 C:03+2] labs: add infra-tracing-nfs account [labs/private] - 10https://gerrit.wikimedia.org/r/1210591 (https://phabricator.wikimedia.org/T399313) (owner: 10Volans) [12:08:30] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:02:57] (03merge) 10wikigit: Add Server Object Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/3 (owner: 10jaredblumer) [13:09:15] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:14:00] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:14:30] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:19:30] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:20:30] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:25:30] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:40:24] (03CR) 10Filippo Giunchedi: [C:03+1] labs: enable infra-tracing-nfs tracing [labs/private] - 10https://gerrit.wikimedia.org/r/1210664 (https://phabricator.wikimedia.org/T399313) (owner: 10Volans) [14:21:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:26:28] FIRING: [9x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-13 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:31:28] FIRING: [21x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-12 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:36:28] FIRING: [37x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-12 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:37:47] 10VPS-project-Codesearch, 06collaboration-services: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11405708 (10Aklapper) Thank you! I'll check https://phabricator.wikimedia.org/diffusion/pulllog/?repo... [14:41:28] FIRING: [46x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-12 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:46:28] FIRING: [55x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-12 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:48:30] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:51:28] FIRING: [63x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:53:02] 06cloud-services-team, 06Wikimedia Enterprise: Toolforge no longer has IP-based access to Wikimedia Enterprise - https://phabricator.wikimedia.org/T410994#11405770 (10taavi) As far as I know any access rules here are something that's set up on the Enterprise end without any setup on the Toolforge infrastructure. [14:53:24] 06cloud-services-team, 06Wikimedia Enterprise, 10Wikimedia Enterprise Volunteer Request: Toolforge no longer has IP-based access to Wikimedia Enterprise - https://phabricator.wikimedia.org/T410994#11405775 (10taavi) [14:56:01] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D visual audit remaining host migrations - https://phabricator.wikimedia.org/T411025#11405784 (10Jclark-ctr) [14:56:28] FIRING: [63x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:01:28] FIRING: [63x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:06:28] FIRING: [60x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:11:28] FIRING: [42x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:16:28] RESOLVED: [30x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [15:18:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [15:19:09] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [15:24:04] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [harbor,infra] Find a way to manage toolforge project policies with code - https://phabricator.wikimedia.org/T360509#11405916 (10fnegri) 05In progress→03Stalled [15:24:51] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [harbor,infra] Find a way to manage toolforge project policies with code - https://phabricator.wikimedia.org/T360509#11405918 (10taavi) What is this stalled on? [15:26:19] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [docs] Update all toolforge repos in gitlab with contribution guidelines and license - https://phabricator.wikimedia.org/T408783#11405924 (10fnegri) 05In progress→03Resolved [15:28:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [15:28:19] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [15:28:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [15:29:31] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [15:48:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [15:52:54] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [16:01:01] 06cloud-services-team, 10Quarry, 07CSS, 13Patch-For-Review: Improve maintenance message CSS - https://phabricator.wikimedia.org/T343644#11406124 (10Aklapper) @framawiki: Would you like to merge your patch? [16:03:44] (03PS3) 10Eugene233: Add Commons base file path to config file [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/954926 (https://phabricator.wikimedia.org/T312178) [16:05:33] (03PS1) 10Jcrespo: garage: Add sample private tokens for non production hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1211160 (https://phabricator.wikimedia.org/T410020) [16:14:34] (03Abandoned) 10Volans: labs: enable infra-tracing-nfs tracing [labs/private] - 10https://gerrit.wikimedia.org/r/1210664 (https://phabricator.wikimedia.org/T399313) (owner: 10Volans) [16:19:59] (03CR) 10Jcrespo: [V:03+2 C:03+2] garage: Add sample private tokens for non production hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1211160 (https://phabricator.wikimedia.org/T410020) (owner: 10Jcrespo) [16:35:32] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] crash recovery can fail because of insufficient innodb_log_file_size - https://phabricator.wikimedia.org/T409922#11406374 (10fnegri) I restarted MariaDB in the replica tools-db-7 to apply the new setting `in... [16:52:44] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D visual audit remaining host migrations - https://phabricator.wikimedia.org/T411025#11406468 (10Jclark-ctr) T405296 these servers are due for refresh and have procurement ticket for replacements already in processes. clouddb1017 clouddb1... [17:00:41] 10Tool-bash: Migrate Bash git hosting from github to gitlab - https://phabricator.wikimedia.org/T342919#11406509 (10bd808) a:05bd808→03None [17:16:02] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] crash recovery can fail because of insufficient innodb_log_file_size - https://phabricator.wikimedia.org/T409922#11406601 (10fnegri) I created a new debugging dashboard in grafana: https://grafana.wmcloud.or... [17:21:00] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 10Wiki-Loves-Monuments-Database, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11406616 (10fnegri) I created a new debugging dashboard in grafana: https://grafana.wmcloud.org/goto/... [17:48:10] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS (Project-requests), 13Patch-For-Review: CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11406711 (10fnegri) 05Open→03In progress a:05dcaro→03fnegri @komla attempted to create this project, but the cookbook failed because of... [18:24:12] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D visual audit remaining host migrations - https://phabricator.wikimedia.org/T411025#11406891 (10Jclark-ctr) [18:55:17] (03PS1) 10Krinkle: write_config: Add operations/software/thumbor-plugins [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1211205 [18:55:52] (03CR) 10Krinkle: [C:03+2] write_config: Add operations/software/thumbor-plugins [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1211205 (owner: 10Krinkle) [18:56:30] (03CR) 10CI reject: [V:04-1] write_config: Add operations/software/thumbor-plugins [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1211205 (owner: 10Krinkle) [19:01:38] 10Tool-Global-user-contributions: Replace rc_type with rc_source - https://phabricator.wikimedia.org/T403710#11407040 (10MarcoAurelio) [19:09:01] (03PS2) 10Krinkle: write_config: Add wikimedia/discovery/ repos to "MediaWiki & services at WMF" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/828055 (owner: 10Chad) [19:09:59] (03CR) 10CI reject: [V:04-1] write_config: Add wikimedia/discovery/ repos to "MediaWiki & services at WMF" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/828055 (owner: 10Chad) [19:25:08] (03PS2) 10Krinkle: write_config: Add operations/software/thumbor-plugins [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1211205 [19:25:13] (03CR) 10Krinkle: [C:03+2] write_config: Add operations/software/thumbor-plugins [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1211205 (owner: 10Krinkle) [19:25:16] (03PS3) 10Krinkle: write_config: Add wikimedia/discovery/ repos to "MediaWiki & services at WMF" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/828055 (owner: 10Chad) [19:25:28] (03CR) 10Krinkle: write_config: Add wikimedia/discovery/ repos to "MediaWiki & services at WMF" (031 comment) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/828055 (owner: 10Chad) [19:26:10] (03Merged) 10jenkins-bot: write_config: Add operations/software/thumbor-plugins [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1211205 (owner: 10Krinkle) [20:02:07] 10VPS-project-Codesearch: Codesearch "everything" index stuck in pre-start state; unable to search "everything" - https://phabricator.wikimedia.org/T410310#11407237 (10Krinkle) 05Resolved→03Open While it may be online, it now stale because we can't apply updates anymore. `counterexample root@codesearch9:/sr... [20:02:37] 10VPS-project-Codesearch: Codesearch "everything" index stuck in pre-start state; unable to search "everything" - https://phabricator.wikimedia.org/T410310#11407239 (10Krinkle) p:05Triage→03High [20:03:26] 10VPS-project-Codesearch: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T410310#11407241 (10Krinkle) [20:03:53] 10VPS-project-Codesearch: Codesearch "everything" index stuck in pre-start state; unable to search "everything" - https://phabricator.wikimedia.org/T410310#11407242 (10Krinkle) 05Open→03Resolved p:05High→03Triage [20:04:46] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047 (10Krinkle) 03NEW [20:06:30] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407259 (10Krinkle) [20:09:26] 06cloud-services-team, 10Cloud-VPS: Cloud VPS alerts for low disk space - https://phabricator.wikimedia.org/T337422#11407267 (10Krinkle) Perhaps an earlier threshold than 10% free because at 86% use (14% free), Linux already can't create new files, somehow? >>! From {T411047} > ` > root@codesearch9:/srv/codes... [20:09:35] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407270 (10Krinkle) [20:11:02] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407273 (10Krinkle) [20:12:36] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407278 (10taavi) You're out of inodes: `lang=shell-session root@codesearch9:~# df -hi /srv Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda... [20:14:22] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407281 (10Krinkle) [20:22:34] 06cloud-services-team, 10Cloud-VPS: Cloud VPS alerts for low disk space - https://phabricator.wikimedia.org/T337422#11407335 (10Volans) The problem is not the space, that disk is out of inodes: ` # df -hi /srv/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda 5.0M 5.0M 0 100% /srv ` I... [20:27:14] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407338 (10Krinkle) I could have sworn that, bugs aside, we have genuinely increased the codesearch disk size several times before, but I can't seem to find any... [20:38:53] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D visual audit remaining host migrations - https://phabricator.wikimedia.org/T411025#11407365 (10RobH) [20:39:15] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D cloud hosts pending migration - https://phabricator.wikimedia.org/T411025#11407366 (10RobH) p:05Triage→03High [20:43:05] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): eqiad row C/D cloud hosts pending migration - https://phabricator.wikimedia.org/T411025#11407370 (10RobH) a:03Andrew I prefer we not wait for the entire refresh of pending Q2 hosts but instead migrate al... [20:52:26] 10Tool-erinnermich: [ErinnerMich] Add a blocklist for sites where requests should be rejected - https://phabricator.wikimedia.org/T410993#11407415 (10Tkarcher) Possible alternative approach: # Pages not suitable for reminders could be assigned to a hidden category. For the AWW pages, this category could be added... [21:08:38] 06cloud-services-team, 10Cloud-VPS, 10VPS-project-Wikistats: cloud VPS - puppet broken - cinderutils 'fstype must not be an empty string' - https://phabricator.wikimedia.org/T410936#11407474 (10Dzahn) p:05Triage→03Low can't explain it and it was an annoyance but not sure how we would even debug this further [21:14:49] (03open) 10raymond-ndibe: images: cache images retrieved from harbor [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/254 [21:15:12] (03update) 10raymond-ndibe: images: cache images retrieved from harbor [repos/cloud/toolforge/jobs-api] (allow_harbor_prebuilt) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/254 [21:15:23] (03update) 10raymond-ndibe: images: cache images retrieved from harbor [repos/cloud/toolforge/jobs-api] (allow_harbor_prebuilt) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/254 [21:18:59] 06cloud-services-team, 10Cloud-VPS: Add default Cloud VPS project alerts for low disk space and low inode count - https://phabricator.wikimedia.org/T337422#11407510 (10bd808) [21:18:59] (03update) 10raymond-ndibe: testing without harbor cache [repos/cloud/toolforge/jobs-api] (allow_harbor_prebuilt) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/254 [21:19:58] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407522 (10Dzahn) The instance codesearch9 is of type `g4.cores4.ram8.disk20`. The part that it has 80G and not just 20 and the name in `/dev/mapper/vd-second--... [21:23:58] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407530 (10Dzahn) While there is a "resize instance" button in Horizon, this allows for selecting another instance flavor. None of the flavors have disks over 2... [21:37:54] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407583 (10Krinkle) https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances#Extend_a_volume {F70655985 height=210} [23:18:08] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407824 (10Dzahn) Oh, yes, of course you are right. Not sure how I did not see it, thanks! Yea, so..next step is the "quota exceeded". Let me create a request... [23:31:09] 10Cloud-VPS (Quota-requests): request to increase disk/volume quota for codesearch project - https://phabricator.wikimedia.org/T411062 (10Dzahn) 03NEW [23:31:50] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407877 (10Dzahn) quota request created -> T411062 per: https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_instances#Increase_quotas_for_projects https://pha... [23:34:22] 10VPS-project-Codesearch, 06collaboration-services: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047#11407884 (10Dzahn) >>! In T411047#11407583, @Krinkle wrote: > https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances#Extend_a_volume... [23:35:31] 10Cloud-VPS (Quota-requests): request to increase disk/volume quota for codesearch project - https://phabricator.wikimedia.org/T411062#11407886 (10Dzahn) [23:45:53] 06cloud-services-team, 10Cloud-VPS (Quota-requests): request to increase disk/volume quota for codesearch project - https://phabricator.wikimedia.org/T411062#11407910 (10bd808) +1