[00:14:50] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [00:15:44] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29699 bytes in 3.261 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [00:16:01] 10Cloud-VPS (Quota-requests): Request quota increase for huma project - https://phabricator.wikimedia.org/T370010 (10Ladsgroup) 03NEW [01:53:50] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [01:56:44] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29698 bytes in 2.501 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [03:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:33:23] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9980124 (10Marostegui) @fnegri before clouddb1021 gets decommissioned - it could be a good test for you to reimage it to bookworm and see how the proc... [06:25:58] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [06:26:50] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29689 bytes in 1.384 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [07:01:39] 10Data-Services, 06Data-Persistence, 13Patch-For-Review: Bring an-redacteddb1001 into service to replace clouddb1021 - https://phabricator.wikimedia.org/T365453#9980171 (10Marostegui) @BTullis we probably need to add an-redacteddb1001 to `hieradata/regex.yaml` somewhere. [07:44:57] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9980242 (10Marostegui) [07:49:56] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:55:41] (03merge) 10aborrero: webhook: don't mutate pods on UPDATE operations [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/13 (https://phabricator.wikimedia.org/T369890) [07:59:19] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: volume-admission: bump to 0.0.51-20240715075554-8f9d4061 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/414 (https://phabricator.wikimedia.org/T369890) [07:59:55] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission [08:00:05] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission [08:02:25] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission [08:02:35] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission [08:05:03] (03update) 10dcaro: toolforge: add webservice configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/127 [08:06:42] (03merge) 10aborrero: volume-admission: bump to 0.0.51-20240715075554-8f9d4061 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/414 (https://phabricator.wikimedia.org/T369890) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:06:55] (03update) 10dcaro: toolforge: add webservice configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/127 [08:06:59] (03merge) 10dcaro: toolforge: add webservice configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/127 [08:15:32] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: kubernetes fails to handle some pods that are being mutated by our admission controllers - https://phabricator.wikimedia.org/T369890#9980270 (10dcaro) There's still the question of what is suddenly updating pods? * If... [08:29:00] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Request quota increase for spacemedia project - https://phabricator.wikimedia.org/T370004#9980301 (10dcaro) +1 [08:29:23] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Request quota increase for spacemedia project - https://phabricator.wikimedia.org/T370004#9980303 (10dcaro) Hmm, interesting project, we might benefit from something like that for toolforge xd [08:31:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: kubernetes fails to handle some pods that are being mutated by our admission controllers - https://phabricator.wikimedia.org/T369890#9980306 (10aborrero) >>! In T369890#9980270, @dcaro wrote: > There's still the quest... [08:31:35] (03merge) 10dcaro: buildpack.toml: Fix homepage url typo [repos/cloud/toolforge/buildpacks/apt-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/apt-buildpack/-/merge_requests/3 (owner: 10bd808) [08:33:26] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: kubernetes fails to handle some pods that are being mutated by our admission controllers - https://phabricator.wikimedia.org/T369890#9980307 (10dcaro) > the newer kubernetes version somehow has additional steps when r... [08:41:45] 10Data-Services, 06Data-Persistence, 13Patch-For-Review: Bring an-redacteddb1001 into service to replace clouddb1021 - https://phabricator.wikimedia.org/T365453#9980339 (10BTullis) >>! In T365453#9980171, @Marostegui wrote: > @BTullis we probably need to add an-redacteddb1001 to `hieradata/regex.yaml` so... [08:43:45] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: kubernetes fails to handle some pods that are being mutated by our admission controllers - https://phabricator.wikimedia.org/T369890#9980343 (10aborrero) >>! In T369890#9980307, @dcaro wrote: >> the newer kubernetes v... [08:45:34] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Request quota increase for spacemedia project - https://phabricator.wikimedia.org/T370004#9980349 (10Slst2020) a:03Slst2020 [08:48:16] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9980358 (10BTullis) >>! In T365424#9980124, @Marostegui wrote: > @fnegri before clouddb1021 gets decommissioned - it could be a good test for you to r... [08:52:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: kubernetes fails to handle some pods that are being mutated by our admission controllers - https://phabricator.wikimedia.org/T369890#9980373 (10dcaro) > In lima-kilo we don't upgrade in place using kubeadm. We rebuild... [08:53:06] 10Cloud-VPS (Quota-requests): Request quota increase for huma project - https://phabricator.wikimedia.org/T370010#9980382 (10dcaro) +1 [09:13:24] 06cloud-services-team, 10Data-Services, 06Data-Platform, 13Patch-For-Review: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#9980452 (10fnegri) p:05Triage→03Medium [09:13:33] (03open) 10dcaro: README: add instruction on how to update the builder/runner [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/53 (https://phabricator.wikimedia.org/T369840) [09:13:43] 10Toolforge, 13Patch-For-Review: `toolforge build run ...` can fail due to docker.io image pull rate limits - https://phabricator.wikimedia.org/T369840#9980456 (10dcaro) [09:14:05] 10Cloud-VPS (Quota-requests): Request quota increase for huma project - https://phabricator.wikimedia.org/T370010#9980458 (10Slst2020) a:03Slst2020 [09:16:42] (03open) 10dcaro: make run image configurable [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/54 [09:25:33] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: `toolforge build run ...` can fail due to docker.io image pull rate limits - https://phabricator.wikimedia.org/T369840#9980501 (10dcaro) p:05Triage→03High a:03dcaro [09:25:46] !log sstefanova@cloudcumin1001 spacemedia START - Cookbook wmcs.openstack.quota_increase [09:25:54] !log sstefanova@cloudcumin1001 spacemedia END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [09:26:16] 10Data-Services: Simple query scans entire revision table on new replicas - https://phabricator.wikimedia.org/T286328#9980510 (10fnegri) p:05Triage→03Low [09:28:39] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: `toolforge build run ...` can fail due to docker.io image pull rate limits - https://phabricator.wikimedia.org/T369840#9980504 (10dcaro) [09:28:49] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: `toolforge build run ...` can fail due to docker.io image pull rate limits - https://phabricator.wikimedia.org/T369840#9980506 (10dcaro) 05Open→03In progress [09:29:27] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: `toolforge jobs` requires current user to be the tool user and listed in NSS passwd data - https://phabricator.wikimedia.org/T369573#9980508 (10dcaro) 05In progress→03Resolved [09:29:48] !log sstefanova@cloudcumin1001 huma START - Cookbook wmcs.openstack.quota_increase [09:29:56] !log sstefanova@cloudcumin1001 huma END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [09:30:36] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering: Raw IPs of logged-out users disclosed in wiki-replicas - https://phabricator.wikimedia.org/T284948#9980515 (10fnegri) [09:30:58] !log sstefanova@cloudcumin1001 spacemedia START - Cookbook wmcs.openstack.quota_increase [09:31:06] !log sstefanova@cloudcumin1001 spacemedia END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [09:34:00] (03update) 10dcaro: README: add instruction on how to update the builder/runner [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/53 (https://phabricator.wikimedia.org/T369840) [09:34:03] (03update) 10dcaro: make run image configurable [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/54 [09:34:05] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering: Raw IPs of logged-out users disclosed in wiki-replicas - https://phabricator.wikimedia.org/T284948#9980518 (10fnegri) p:05Triage→03Low > I am fine with having this ticket stalled until IP masking (T283177) is effective,... [09:36:13] 10Cloud-VPS (Quota-requests): Request quota increase for huma project - https://phabricator.wikimedia.org/T370010#9980524 (10Slst2020) 05Open→03Resolved Done – hope this solves your problem! [09:36:51] !log sstefanova@cloudcumin1001 spacemedia START - Cookbook wmcs.openstack.quota_increase [09:36:59] !log sstefanova@cloudcumin1001 spacemedia END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [09:40:21] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Request quota increase for spacemedia project - https://phabricator.wikimedia.org/T370004#9980528 (10Slst2020) 05Open→03Resolved Done! [09:42:17] 10Data-Services: Optimize querying the page table by namespace - https://phabricator.wikimedia.org/T252122#9980547 (10fnegri) p:05Triage→03Low [09:43:29] 10Data-Services: Aggregate query on page, revision and langlinks takes a long time to run - https://phabricator.wikimedia.org/T252356#9980551 (10fnegri) p:05Triage→03Low [09:43:54] 10Data-Services: Query is too slow ever since the migration to actor table - https://phabricator.wikimedia.org/T251801#9980554 (10fnegri) p:05Triage→03Low [09:44:05] (03approved) 10sstefanova: make run image configurable [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/54 (owner: 10dcaro) [09:44:41] 10Data-Services, 13Patch-For-Review, 10Wiki-Setup (Create): Create a Wikimedians of United Arab Emirates User Group Wiki - https://phabricator.wikimedia.org/T362529#9980553 (10BTullis) >>! In T362529#9972354, @ABran-WMF wrote: > @Zabe it seems we were missing the "storage layer" task we usually get. Anyway,... [09:59:35] 06cloud-services-team, 10Data-Services, 06Data-Engineering-Icebox: Mitigate breaking changes from the new Wiki Replicas architecture - https://phabricator.wikimedia.org/T280152#9980570 (10fnegri) 05Open→03Resolved a:03fnegri Marking this as Resolved as all the main subtasks have been completed. The... [10:08:18] (03update) 10dcaro: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [10:12:10] 10Data-Services, 13Patch-For-Review, 10Wiki-Setup (Create): Create a Wikimedians of United Arab Emirates User Group Wiki - https://phabricator.wikimedia.org/T362529#9980631 (10ABran-WMF) >>! In T362529#9980552, @BTullis wrote: > @ABran-WMF, @Fnegri - I believe that we are ready to run the `sre.wikireplicas... [10:16:49] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "petscan" project Buster deprecation - https://phabricator.wikimedia.org/T367545#9980665 (10Magnus) Hello, didn't see this until now. Happy to move to to a new VM. Can you provide (and throw in a bit more disk space please?), or should I do this myself? petscan... [10:22:13] (03open) 10aborrero: k8s: registry-admission: wait when deploying [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/168 (https://phabricator.wikimedia.org/T369527) [10:41:21] (03update) 10dcaro: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [10:44:52] 10Cloud-VPS, 07Epic: CloudVPS: introduce tenant networks - https://phabricator.wikimedia.org/T270694#9980769 (10aborrero) [10:44:56] (03update) 10dcaro: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [10:46:53] (03approved) 10dcaro: k8s: registry-admission: wait when deploying [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/168 (https://phabricator.wikimedia.org/T369527) (owner: 10aborrero) [10:47:21] (03close) 10dcaro: README: add instruction on how to update the builder/runner [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/53 (https://phabricator.wikimedia.org/T369840) [10:47:38] (03approved) 10dcaro: make run image configurable [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/54 [10:47:39] (03update) 10dcaro: make run image configurable [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/54 [10:47:52] 06cloud-services-team: SystemdUnitDown Unit opentofu-infra-diff.service on node cloudcontrol1007 has been down for long. - https://phabricator.wikimedia.org/T367263#9980776 (10aborrero) 05Open→03Resolved a:03aborrero alert went away, we don't have more info. Taavi is no longer around. Closing now, fee... [10:48:21] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "petscan" project Buster deprecation - https://phabricator.wikimedia.org/T367545#9980779 (10Magnus) Update: Building new VM now [10:49:45] (03merge) 10dcaro: make run image configurable [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/54 [10:51:40] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.112-20240715104956-20e1f392 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/415 (https://phabricator.wikimedia.org/T369840) [10:55:30] (03update) 10dcaro: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [10:55:52] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "petscan" project Buster deprecation - https://phabricator.wikimedia.org/T367545#9980814 (10Aklapper) >>! In T367545#9980665, @Magnus wrote: > Can you provide (and throw in a bit more disk space please?), or should I do this myself? For quota requests, see http... [10:58:35] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "petscan" project Buster deprecation - https://phabricator.wikimedia.org/T367545#9980818 (10Magnus) I have * deleted petscan4 * deleted petscan4a * created petscan5 (as debian-12.0-bookworm) * created a new web proxy to point to petscan5 (only seens to offer ht... [11:01:29] (03merge) 10aborrero: k8s: registry-admission: wait when deploying [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/168 (https://phabricator.wikimedia.org/T369527) [11:12:29] (03update) 10dcaro: builds-builder: bump to 0.0.112-20240715104956-20e1f392 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/415 (https://phabricator.wikimedia.org/T369840) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:23:50] 06cloud-services-team, 07Epic: Cloud VPS: consider extending tofu-infra coverage - https://phabricator.wikimedia.org/T370037 (10aborrero) 03NEW [11:23:53] (03open) 10dcaro: pipeline: fix variable typo [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/55 [11:24:37] 06cloud-services-team: tofu-infra: enable for codfw1dev - https://phabricator.wikimedia.org/T370038 (10aborrero) 03NEW [11:28:20] (03approved) 10dcaro: pipeline: fix variable typo [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/55 [11:28:35] (03merge) 10dcaro: pipeline: fix variable typo [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/55 [11:28:35] 06cloud-services-team: tofu-infra: enable for codfw1dev - https://phabricator.wikimedia.org/T370038#9980891 (10aborrero) this was enabled already. What I will do is to document how it works here https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/OpenTofu [11:29:58] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.112-20240715104956-20e1f392 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/415 (https://phabricator.wikimedia.org/T369840) [11:30:21] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [11:31:13] 06cloud-services-team, 07Epic: Cloud VPS: consider extending tofu-infra coverage - https://phabricator.wikimedia.org/T370037#9980893 (10aborrero) [11:31:53] (03update) 10dcaro: builds-builder: bump to 0.0.113-20240715112850-70f3503e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/415 (https://phabricator.wikimedia.org/T369840) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:33:13] (03update) 10dcaro: builds-builder: bump to 0.0.113-20240715112850-70f3503e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/415 (https://phabricator.wikimedia.org/T369840) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:33:15] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29689 bytes in 2.431 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [11:35:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [11:35:39] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [11:37:41] 06cloud-services-team: tofu-infra: enable for codfw1dev - https://phabricator.wikimedia.org/T370038#9980906 (10aborrero) 05Open→03Invalid [11:40:01] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [11:40:12] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [11:45:13] (03approved) 10dcaro: builds-builder: bump to 0.0.113-20240715112850-70f3503e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/415 (https://phabricator.wikimedia.org/T369840) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:45:17] (03merge) 10dcaro: builds-builder: bump to 0.0.113-20240715112850-70f3503e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/415 (https://phabricator.wikimedia.org/T369840) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:45:38] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: `toolforge build run ...` can fail due to docker.io image pull rate limits - https://phabricator.wikimedia.org/T369840#9980921 (10dcaro) [11:48:16] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: `toolforge build run ...` can fail due to docker.io image pull rate limits - https://phabricator.wikimedia.org/T369840#9980923 (10dcaro) 05In progress→03Resolved [11:49:56] FIRING: CloudVPSDesignateLeaks: Detected 10 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:23:51] (03approved) 10sstefanova: webhook: don't mutate pods on UPDATE operations [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/8 (https://phabricator.wikimedia.org/T369890) (owner: 10aborrero) [12:25:05] (03update) 10aborrero: webhook: don't mutate pods on UPDATE operations [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/8 (https://phabricator.wikimedia.org/T369890) [12:25:06] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/15 [12:25:13] (03merge) 10aborrero: webhook: don't mutate pods on UPDATE operations [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/8 (https://phabricator.wikimedia.org/T369890) [12:26:28] (03open) 10sstefanova: api: rename api resources to plural [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T365014) [12:26:32] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1054331 (owner: 10L10n-bot) [12:26:35] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1054335 (owner: 10L10n-bot) [12:26:36] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1054336 (owner: 10L10n-bot) [12:27:52] (03update) 10sstefanova: api: rename api resources to plural [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T365014) [12:31:48] (03open) 10aborrero: tofu-infra: introduce Cloud VPS networks for codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/13 (https://phabricator.wikimedia.org/T370037) [12:37:52] (03open) 10aborrero: README: include pointers to documents [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/14 [12:38:24] (03merge) 10aborrero: README: include pointers to documents [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/14 [12:39:21] (03update) 10aborrero: tofu-infra: introduce Cloud VPS networks for codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/13 (https://phabricator.wikimedia.org/T370037) [12:39:37] 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9981080 (10dcaro) 05Open→03Resolved a:03dcaro [12:42:38] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.1 Consulting Toolforge roots/maintainers - https://phabricator.wikimedia.org/T368601#9981095 (10dcaro) 05Open→03In progress [12:42:50] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.2 Create "standard" tool to measure the number of steps for a deployment - https://phabricator.wikimedia.org/T368602#9981091 (10dcaro) 05Open→03In progress [12:43:19] (03update) 10aborrero: tofu-infra: introduce Cloud VPS networks for codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/13 (https://phabricator.wikimedia.org/T370037) [12:43:23] (03update) 10sstefanova: api: remove /api prefix [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 (https://phabricator.wikimedia.org/T365014) [12:43:46] (03update) 10sstefanova: remove /api prefix [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/50 [12:43:54] (03update) 10sstefanova: remove /api prefix [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/50 [12:43:57] (03update) 10aborrero: tofu-infra: introduce Cloud VPS networks for codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/13 (https://phabricator.wikimedia.org/T370037) [12:44:13] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [builds-api,envvars-api,jobs-api] bump the version in the openapi definition when bumping the package version - https://phabricator.wikimedia.org/T356972#9981089 (10dcaro) 05Open→03Resolved [12:47:08] (03update) 10sstefanova: api: rename api resources to plural [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T365014) [12:54:11] (03update) 10aborrero: tofu-infra: introduce Cloud VPS networks for codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/13 (https://phabricator.wikimedia.org/T370037) [13:02:51] (03update) 10aborrero: tofu-infra: introduce Cloud VPS networks for codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/13 (https://phabricator.wikimedia.org/T370037) [13:11:37] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26 - https://phabricator.wikimedia.org/T327025#9981169 (10dcaro) [13:18:16] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26 - https://phabricator.wikimedia.org/T327025#9981189 (10dcaro) [13:21:33] 06cloud-services-team, 10Toolforge: [infra,k8s] Move to kubernetes PAVs and drop kyverno - https://phabricator.wikimedia.org/T364293#9981196 (10dcaro) The validating admission policy is not stable until 1.30 (1.26/27 -> beta, 1.28/29 -> alpha, 1.30 -> stable) [13:21:42] 06cloud-services-team, 10Toolforge: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046 (10Slst2020) 03NEW [13:22:14] 10Toolforge: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#9981211 (10dcaro) [13:22:15] 06cloud-services-team, 10Toolforge: [infra,k8s] Move to kubernetes PAVs and drop kyverno - https://phabricator.wikimedia.org/T364293#9981210 (10dcaro) [13:22:17] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26 - https://phabricator.wikimedia.org/T327025#9981212 (10dcaro) [13:22:23] 06cloud-services-team, 10Toolforge: [infra,k8s] Move to kubernetes PAVs and drop kyverno - https://phabricator.wikimedia.org/T364293#9981213 (10dcaro) [13:23:50] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26 - https://phabricator.wikimedia.org/T327025#9981220 (10dcaro) [13:23:54] 06cloud-services-team, 10Toolforge: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9981221 (10Slst2020) [13:25:15] 06cloud-services-team, 10Toolforge: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9981228 (10Slst2020) a:05Slst2020→03None [13:26:22] (03update) 10dcaro: openapi: consolidate metrics and healthz endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T365014) (owner: 10sstefanova) [13:30:36] 06cloud-services-team, 10Toolforge: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9981258 (10Slst2020) [13:30:49] 06cloud-services-team, 10Toolforge: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9981260 (10Slst2020) [13:31:03] (03approved) 10dcaro: openapi: consolidate metrics and healthz endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T365014) (owner: 10sstefanova) [13:31:27] 06cloud-services-team, 10Toolforge: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9981264 (10Slst2020) [13:50:49] (03update) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T367181) [13:52:42] (03open) 10dcaro: functional: allow running on two different tools at a time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/416 [13:53:13] (03open) 10sstefanova: api endpoints: use plural paths [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/51 (https://phabricator.wikimedia.org/T365014) [13:57:21] (03merge) 10sstefanova: openapi: consolidate metrics and healthz endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T365014) [14:00:07] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.315-20240715135730-fb3bd3e7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/417 (https://phabricator.wikimedia.org/T365014) [14:05:25] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [14:06:02] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [14:07:10] (03update) 10sstefanova: functional: allow running on two different tools at a time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/416 (owner: 10dcaro) [14:07:12] (03approved) 10sstefanova: functional: allow running on two different tools at a time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/416 (owner: 10dcaro) [14:07:20] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [14:07:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:07:32] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) [14:07:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:08:13] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [14:08:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:13:51] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [14:13:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:15:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:16:48] (03approved) 10dcaro: functional: allow running on two different tools at a time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/416 [14:16:51] (03merge) 10dcaro: functional: allow running on two different tools at a time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/416 [14:20:02] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [14:24:45] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [14:28:51] (03update) 10sstefanova: api endpoints: use plural paths [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/51 (https://phabricator.wikimedia.org/T365014) [14:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:30:09] (03update) 10sstefanova: jobs-api: bump to 0.0.315-20240715135730-fb3bd3e7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/417 (https://phabricator.wikimedia.org/T365014) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:33:23] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [14:33:34] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [14:34:08] (03update) 10aborrero: tofu-infra: introduce Cloud VPS networks for codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/13 (https://phabricator.wikimedia.org/T370037) [14:41:13] 06cloud-services-team, 10Cloud-VPS, 10Cumin, 06Infrastructure-Foundations: Cumin: create external backend for WMCS Puppet API - https://phabricator.wikimedia.org/T179816#9981528 (10fnegri) [14:42:38] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [14:42:49] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [14:55:58] (03update) 10sstefanova: jobs-api: bump to 0.0.315-20240715135730-fb3bd3e7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/417 (https://phabricator.wikimedia.org/T365014) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:56:26] (03update) 10aborrero: tofu-infra: introduce Cloud VPS networks for codfw1dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/13 (https://phabricator.wikimedia.org/T370037) [14:56:39] (03merge) 10sstefanova: jobs-api: bump to 0.0.315-20240715135730-fb3bd3e7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/417 (https://phabricator.wikimedia.org/T365014) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [15:03:49] (03update) 10dcaro: run_functional_tests: when custom tool is passed, set the uid too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/410 [15:03:56] 06cloud-services-team, 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#9981687 (10aborrero) 05Open→03In progress p:05Triage→03High a:03aborrero this task is in the list of focus areas and WMCS team goals for Q1 FY 2024/25 [15:04:03] (03update) 10sstefanova: api: remove /api prefix [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 (https://phabricator.wikimedia.org/T365014) [15:04:24] (03update) 10dcaro: run_functional_tests: when custom tool is passed, set the uid too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/410 [15:06:10] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [15:06:36] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [15:13:14] 06cloud-services-team, 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#9981748 (10aborrero) [15:13:15] 10Cloud-VPS, 07Epic: CloudVPS: introduce tenant networks - https://phabricator.wikimedia.org/T270694#9981749 (10aborrero) [15:23:43] (03update) 10bd808: Make image useful for Brad [toolforge-repos/bd808-buildpack-perl-bastion] - 10https://gitlab.wikimedia.org/toolforge-repos/bd808-buildpack-perl-bastion/-/merge_requests/1 [15:52:24] (03update) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T363346) [16:02:20] (03approved) 10dcaro: run_functional_tests: when custom tool is passed, set the uid too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/410 [16:02:45] (03unapproved) 10dcaro: run_functional_tests: when custom tool is passed, set the uid too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/410 [16:05:35] 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations, 06Release-Engineering-Team: Cloud VPS "integration" project Buster deprecation - https://phabricator.wikimedia.org/T367534#9982228 (10hashar) a:03hashar I will do the last two sets (cumin and pkgbuilder) tomorrow, it is a bit too late f... [16:09:05] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [16:11:08] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [16:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:20:09] 06cloud-services-team, 10Technical-blog-posts: Tech blog post: "Wikimedia Toolforge: migrating Kubernetes from PodSecurityPolicy to kyverno" - https://phabricator.wikimedia.org/T368948#9982274 (10debt) 05Open→03Resolved [16:34:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:47:23] (03update) 10dcaro: registry-admission: add local harbor as allowed registry for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/413 [16:47:24] (03update) 10dcaro: registry-admission: add local harbor as allowed registry for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/413 [16:48:35] (03merge) 10dcaro: registry-admission: add local harbor as allowed registry for lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/413 [16:48:58] 10Cloud-Services, 10Catalyst: Moving proxies across wmcs projects for patchdemo.wmflabs.org - https://phabricator.wikimedia.org/T370080 (10thcipriani) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and r... [16:49:47] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [16:49:51] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [16:50:49] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [16:50:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:53:29] (03update) 10dcaro: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) [16:58:33] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Request quota increase for spacemedia project - https://phabricator.wikimedia.org/T370004#9982451 (10bd808) @Don-vip Please be really careful that you do not collect and expose end-user IP addresses with your GlitchTip and GitLab int... [17:05:41] (03PS1) 10David Caro: depool_and_destroy: also zap the devices [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054376 [17:06:26] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [17:06:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:07:20] (03update) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T367181) [17:10:06] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [17:10:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:18:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [17:33:20] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) [17:33:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:38:00] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Request quota increase for spacemedia project - https://phabricator.wikimedia.org/T370004#9982658 (10Don-vip) @bd808 ok, noted! Thanks a lot @Slst2020! [18:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:23:18] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1054331 (owner: 10L10n-bot) [18:24:17] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1054335 (owner: 10L10n-bot) [18:24:19] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1054336 (owner: 10L10n-bot) [18:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:38:48] (03update) 10sstefanova: api: remove /api prefix [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 (https://phabricator.wikimedia.org/T365014) [18:42:58] (03merge) 10sstefanova: api: remove /api prefix [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 (https://phabricator.wikimedia.org/T365014) [18:42:58] (03update) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T363346) [18:58:39] (03PS2) 10David Caro: depool_and_destroy: also zap the devices [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054376 [18:59:04] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [18:59:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:59:42] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) [18:59:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:21:22] (03update) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T363346) [19:27:44] (03update) 10sstefanova: api: consolidate paths [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T365014) [19:33:42] (03update) 10sstefanova: api: consolidate paths [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T365014) [19:36:39] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [19:39:29] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29691 bytes in 0.190 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [19:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:24:03] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [20:24:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:29:05] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [20:29:07] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [20:29:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:29:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:40:46] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) [20:40:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:44:39] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [20:46:31] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29705 bytes in 1.310 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [21:22:16] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:27:16] RESOLVED: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:33:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [22:33:34] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [22:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks