[02:12:21] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 10Ceph, 06DC-Ops, and 2 others: [cloudceph] test the new DELL hard drives throughput - https://phabricator.wikimedia.org/T390134#10728390 (10Andrew) a:05Andrew→03dcaro After a bit of monkeying with the raid settings (to mark the new drive as non-r... [04:33:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:43:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:06:35] !log andrew@cloudcumin1001 math START - Cookbook wmcs.openstack.quota_increase (T390656) [05:06:39] T390656: Increase disk quota for math - https://phabricator.wikimedia.org/T390656 [05:06:49] !log andrew@cloudcumin1001 math END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T390656) [05:07:42] 10Cloud-VPS (Quota-requests): Increase disk quota for math - https://phabricator.wikimedia.org/T390656#10728503 (10Andrew) 05Open→03Resolved a:03Andrew I'm sorry it took so long to fill this completely reasonable request! I've increased your quota to 310GB total. [06:03:52] 06cloud-services-team, 10Cloud-VPS, 10Tool-spacemedia, 10video2commons, 07Upstream: Cloud Services shared IP (static NAT for external communications) often rate limited by YouTube for video downloads - https://phabricator.wikimedia.org/T236446#10728543 (10dbenjaminmiller) The notion that YouTube videos m... [08:26:10] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: improve networktests for newer network setup - https://phabricator.wikimedia.org/T391325#10728869 (10aborrero) refreshed docs at https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Network/Tests [08:28:30] 06cloud-services-team, 10Cloud-VPS: gitlab ci: validate secrets settings in pipeline for tofu integration - https://phabricator.wikimedia.org/T391467#10728875 (10aborrero) >! In T391467#10726025, @aborrero wrote: > I will refresh the openstack API application credentials to be from `srv-networktests` as well,... [08:30:46] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: introduce additional gitlab-ci automation - https://phabricator.wikimedia.org/T370652#10728882 (10aborrero) Another example repository with a full workflow based on gitlab CI/CD: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provi... [08:37:45] (03open) 10aborrero: gitlab-ci: disable tofu apply on schedule pipelines [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/7 (https://phabricator.wikimedia.org/T391325) [08:40:37] (03open) 10aborrero: floating_ip: reference port.id not port.port_id [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/8 (https://phabricator.wikimedia.org/T391325) [08:41:57] (03merge) 10aborrero: floating_ip: reference port.id not port.port_id [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/8 (https://phabricator.wikimedia.org/T391325) [08:42:26] (03update) 10aborrero: gitlab-ci: disable tofu apply on schedule pipelines [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/7 (https://phabricator.wikimedia.org/T391325) [08:47:48] (03merge) 10aborrero: gitlab-ci: disable tofu apply on schedule pipelines [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/7 (https://phabricator.wikimedia.org/T391325) [08:56:19] (03open) 10aborrero: gitlab-ci: reverse rules order for tofu apply step [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/9 [08:57:02] (03merge) 10aborrero: gitlab-ci: reverse rules order for tofu apply step [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/9 [09:11:45] (03open) 10aborrero: networktests_infra: create DNS records for the floating IPs [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/10 (https://phabricator.wikimedia.org/T391325) [09:12:50] (03update) 10aborrero: networktests_infra: create DNS records for the floating IPs [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/10 (https://phabricator.wikimedia.org/T391325) [09:14:57] (03merge) 10aborrero: networktests_infra: create DNS records for the floating IPs [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/10 (https://phabricator.wikimedia.org/T391325) [09:34:36] (03open) 10aborrero: networktests_infra: drop VM instance for dualstack floating IP case [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/11 (https://phabricator.wikimedia.org/T391325) [09:36:01] (03merge) 10aborrero: networktests_infra: drop VM instance for dualstack floating IP case [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/11 (https://phabricator.wikimedia.org/T391325) [09:50:30] (03open) 10aborrero: codfw1dev: add AAAA record for cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/177 (https://phabricator.wikimedia.org/T391325) [09:54:39] (03update) 10aborrero: codfw1dev: add AAAA record for cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/177 (https://phabricator.wikimedia.org/T391325) [09:55:45] (03update) 10aborrero: codfw1dev: add AAAA record for cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/177 (https://phabricator.wikimedia.org/T391325) [09:56:58] (03merge) 10aborrero: codfw1dev: add AAAA record for cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/177 (https://phabricator.wikimedia.org/T391325) [09:56:59] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [09:57:33] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:04:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-76 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [12:24:13] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/16 [12:37:07] (03open) 10aborrero: cloudinfra-codfw1dev: allow puppet-agent to run from the VXLAN / IPv4 subnet [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/178 (https://phabricator.wikimedia.org/T391325) [12:37:51] (03merge) 10aborrero: cloudinfra-codfw1dev: allow puppet-agent to run from the VXLAN / IPv4 subnet [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/178 (https://phabricator.wikimedia.org/T391325) [12:37:58] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:38:33] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:42:17] (03open) 10aborrero: networktests_infra: shorten VM names [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/13 [12:44:56] (03merge) 10aborrero: networktests_infra: shorten VM names [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/13 [12:52:59] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/16 (owner: 10l10n-bot) [12:53:02] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/16 (owner: 10l10n-bot) [13:11:17] 06cloud-services-team, 10Cloud-VPS, 10Tool-spacemedia, 10video2commons, 07Upstream: Cloud Services shared IP (static NAT for external communications) often rate limited by YouTube for video downloads - https://phabricator.wikimedia.org/T236446#10729597 (10bvibber) >>! In T236446#10728114, @Jeff_G wrote:... [13:16:57] 06cloud-services-team, 10Toolforge: [envvars-cli] Enable use of `toolforge envvar` managed data on bastions - https://phabricator.wikimedia.org/T358537#10729637 (10Addshore) [13:19:16] 06cloud-services-team, 10Toolforge: [webservice] should have more easily understandable error messages when run as a non-tool user - https://phabricator.wikimedia.org/T360478#10729642 (10Addshore) A note from the possible future of https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli This CLI... [13:20:35] 06cloud-services-team, 10Toolforge: [envvars-cli] Add option to not show envvar values when listing - https://phabricator.wikimedia.org/T363544#10729649 (10Addshore) [13:20:45] 06cloud-services-team, 10Toolforge: [jobs-api,jobs-cli] Expose hidden quota errors more clearly - https://phabricator.wikimedia.org/T333976#10729651 (10Addshore) [13:21:50] 06cloud-services-team, 10Toolforge: toolforge-cli: Allow configuring description for external subcommands - https://phabricator.wikimedia.org/T336052#10729657 (10Addshore) A note on what the output of https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli looks like ` Toolforge CLI Usage: too... [13:23:44] 06cloud-services-team, 10Toolforge: [builds-cli] No obvious way to delete individual `toolforge build` generated artifacts other than `toolforge clean` - https://phabricator.wikimedia.org/T368317#10729665 (10Addshore) [13:46:15] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcontrol1005.eqiad.wmnet - https://phabricator.wikimedia.org/T391413#10729730 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudcontrol1005.eqiad.wmnet` - cloudcontrol1005.... [13:49:35] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 13Patch-For-Review: decommission cloudcontrol1005.eqiad.wmnet - https://phabricator.wikimedia.org/T391413#10729762 (10Andrew) [13:49:51] 10cloud-services-team (Hardware): cloudcontrol1011 service implementation tracking - https://phabricator.wikimedia.org/T391300#10729765 (10Andrew) 05Open→03Resolved [14:59:29] 10Striker, 13Patch-For-Review: Add Bitu container to Striker development environment - https://phabricator.wikimedia.org/T362318#10730162 (10SLyngshede-WMF) @Arendpieter I believe that the people working on Striker prefers to have the ability to do full end to end testing, within a controlled environment, That... [15:00:29] 06cloud-services-team, 10Cloud-VPS, 07Essential-Work, 10Release-Engineering-Team (Radar): On WMCS linux-perf must be installed from backports to be in sync with linux-image package - https://phabricator.wikimedia.org/T390824#10730183 (10thcipriani) [15:05:04] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: [designate] migrate DNS records in noauth-project to cloudinfra project - https://phabricator.wikimedia.org/T380484#10730210 (10fnegri) 05Open→03In progress a:03fnegri [15:10:58] (03PS8) 10Slyngshede: Add Bitu container [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318) [15:23:39] 06cloud-services-team, 14Toolforge (Toolforge iteration 18), 07Security: toolforge: we may be allowing arbitrary host path mounts in containers - https://phabricator.wikimedia.org/T386921#10730346 (10aborrero) this is fixed by patch: https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/... [15:28:32] (03open) 10aborrero: codfw1dev: testlabs: reduce floating IP quotas to 2 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/179 (https://phabricator.wikimedia.org/T391325) [15:30:45] 06cloud-services-team, 14Toolforge (Toolforge iteration 18), 07Security: toolforge: we may be allowing arbitrary host path mounts in containers - https://phabricator.wikimedia.org/T386921#10730397 (10Andrew) thx arturo :) [15:32:29] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: introduce additional gitlab-ci automation - https://phabricator.wikimedia.org/T370652#10730409 (10aborrero) see also: https://docs.gitlab.com/user/infrastructure/iac/mr_integration/#manually-configure-opentofu-report-artifacts [15:35:10] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 [15:38:27] (03open) 10aborrero: gitlab-ci: show tofu plan in the MR widget [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/14 [15:39:00] (03open) 10fnegri: Draft: import some db-related DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/180 (https://phabricator.wikimedia.org/T374953) [15:39:57] (03close) 10fnegri: tools-db DNS: update comments and descriptions [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/144 (https://phabricator.wikimedia.org/T352206 https://phabricator.wikimedia.org/T374953) [15:40:37] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 [15:40:49] (03update) 10fnegri: Draft: import some db-related DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/180 (https://phabricator.wikimedia.org/T374953) [15:41:01] (03update) 10aborrero: gitlab-ci: show tofu plan in the MR widget [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/14 [15:44:06] (03update) 10aborrero: gitlab-ci: show tofu plan in the MR widget [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/14 [15:46:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-76 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:53:38] (03approved) 10fnegri: gitlab-ci: show tofu plan in the MR widget [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/14 (owner: 10aborrero) [15:59:36] (03merge) 10aborrero: gitlab-ci: show tofu plan in the MR widget [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/14 [16:04:40] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Update scripts that are referencing "noauth-project" for Designate - https://phabricator.wikimedia.org/T391486#10730571 (10fnegri) 05In progress→03Resolved [16:05:34] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: [designate] migrate DNS records in noauth-project to cloudinfra project - https://phabricator.wikimedia.org/T380484#10730579 (10fnegri) All zones were moved to `cloudinfra`. ` fnegri@cloudcontrol1007:~$ sudo wmcs-openstack zone list --sudo-project-id noa... [16:05:51] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: [designate] migrate DNS records in noauth-project to cloudinfra project - https://phabricator.wikimedia.org/T380484#10730583 (10fnegri) 05In progress→03Resolved [16:07:22] 10tool-docs, 10gadget-Cat-a-lot, 06RoadToWiki, 07Documentation: Write a documentation to set up local development environment for Cat-a-lot - https://phabricator.wikimedia.org/T387139#10730593 (10TBurmeister) [16:25:25] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Provide any rough metrics for tool and project usage - https://phabricator.wikimedia.org/T178834#10730673 (10TBurmeister) [16:28:14] 10Tool-documentation, 10gadget-Cat-a-lot, 06RoadToWiki, 07Documentation: Write a documentation to set up local development environment for Cat-a-lot - https://phabricator.wikimedia.org/T387139#10730683 (10TBurmeister) [16:39:24] 06cloud-services-team, 10Cloud-VPS, 10Tool-spacemedia, 10video2commons, 07Upstream: Cloud Services shared IP (static NAT for external communications) often rate limited by YouTube for video downloads - https://phabricator.wikimedia.org/T236446#10730766 (10Yann) >>! In T236446#10728543, @dbenjaminmiller w... [17:07:20] 10Tool-documentation: WMAHack25: Wikimedia Tool Documentation - https://phabricator.wikimedia.org/T390349#10730942 (10TBurmeister) 05In progress→03Resolved [17:13:39] 10Tool-documentation: Curate tool documentation tasks and resources for upcoming Hackathons - https://phabricator.wikimedia.org/T391611 (10TBurmeister) 03NEW [17:14:00] 10Tool-documentation: Curate tool documentation tasks and resources for upcoming Hackathons - https://phabricator.wikimedia.org/T391611#10730977 (10TBurmeister) 05Open→03In progress p:05Triage→03Low [17:14:19] 10Tool-documentation: Curate tool documentation tasks and resources for upcoming Hackathons - https://phabricator.wikimedia.org/T391611#10730983 (10TBurmeister) [17:22:58] 10Tool-documentation, 10Adiutor, 06Wiki-Mentor-Africa, 07good first task: [Documentation] Update and improve docs for Adiutor tool - https://phabricator.wikimedia.org/T390392#10731008 (10TBurmeister) p:05Triage→03Low Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcom... [17:39:11] 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10731074 (10Andrew) Regarding item 1 and 3, I think I my check for detecting redirect pages was too broad and excluded other useful pages. I'm rebui... [18:13:05] 10Tool-wikiloves: Wiki Loves Africa 2025 Stats on Toolforge - https://phabricator.wikimedia.org/T391617 (10Kateregga1) 03NEW [19:40:16] 06cloud-services-team, 10Toolforge, 10Tools: Flickr blocking image requests from Toolforge k8s, breaking multiple tools - https://phabricator.wikimedia.org/T384468#10731385 (10Andrew) 05Open→03Stalled p:05High→03Low [20:33:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:34:45] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Modernize openstack rbac - https://phabricator.wikimedia.org/T330759#10731591 (10Andrew) [20:43:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks