[00:17:35] 10ToolforgeBundle: SSH command should accept a shell username - https://phabricator.wikimedia.org/T365776 (10MusikAnimal) 03NEW [00:18:37] 10ToolforgeBundle: SSH command should accept a shell username - https://phabricator.wikimedia.org/T365776#9828289 (10MusikAnimal) 05Open→03Invalid Bah, the `--username` exists, just isn't mentioned in the README. [00:44:39] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:45:03] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:49:39] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:46:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:45:03] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:29:43] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1035384 (owner: 10L10n-bot) [06:17:24] 10ToolforgeBundle: SSH command should accept a shell username - https://phabricator.wikimedia.org/T365776#9828537 (10Samwilson) Ooh, an [[ https://packagist.org/packages/samwilson/console-readme-generator | automatic CLI command readme generator]] is needed! :-P [06:46:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:46:18] (03PS1) 10Slyngshede: Add Bitu container [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318) [07:48:09] (03PS2) 10Slyngshede: Add Bitu container [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318) [08:11:38] 14Grid-Engine-to-K8s-Migration, 10Wiki-Loves-Monuments-Database: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787#9828665 (10JeanFred) >>! In T319787#9605216, @dcaro wrote: > can you please publish the code for your tool on a public repository (like... [08:43:04] 14Grid-Engine-to-K8s-Migration, 10Wiki-Loves-Monuments-Database: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787#9828729 (10dcaro) >>! In T319787#9828665, @JeanFred wrote: >>>! In T319787#9605216, @dcaro wrote: >> can you please publish the code fo... [08:45:03] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:45:29] 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Upgrade clouddb1021 to bookworm - https://phabricator.wikimedia.org/T365450#9828739 (10Gehel) p:05Triage→03High [09:00:17] 14Grid-Engine-to-K8s-Migration, 10Wiki-Loves-Monuments-Database: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787#9828796 (10JeanFred) Thanks for the answer!: Here’s the toolhub record: https://toolhub.wikimedia.org/tools/heritage-api >>! In T3197... [09:42:50] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [09:58:44] 10Cloud Services Proposals: Decision request - What to use for toolforge components api task execution - https://phabricator.wikimedia.org/T362224#9828902 (10aborrero) I think either options 2 or 3 are fine for me. With the assumptions that: * option 2 is maybe faster to implement today * option 3 is more robus... [10:04:25] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons: Adding new members to Cloud VPS project fails - https://phabricator.wikimedia.org/T365096#9828931 (10Zache) Is there any timetable when this is fixed (or is there alternative methods to add project members)? [10:46:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:52:44] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [11:19:19] 14Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9829083 (10MBH) >>! In T320164#9635401, @MBH wrote: > This bots has also a strange issue: they write messages about correct working into error stream, see conten... [11:45:40] (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110) [12:27:24] (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110) [12:45:04] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [13:51:02] 10Cloud-VPS: object store for tf-infra-test - https://phabricator.wikimedia.org/T365820 (10rook) 03NEW [13:55:26] 10Cloud-VPS: object store for tf-infra-test - https://phabricator.wikimedia.org/T365820#9829720 (10rook) [14:24:08] 10Cloud-VPS (Project-requests): TfInfraTest project - https://phabricator.wikimedia.org/T365822 (10rook) 03NEW [14:24:31] 10Cloud-VPS: object store for tf-infra-test - https://phabricator.wikimedia.org/T365820#9829792 (10rook) [14:24:33] 10Cloud-VPS (Project-requests): TfInfraTest project - https://phabricator.wikimedia.org/T365822#9829793 (10rook) [14:26:42] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 10 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:50:32] 10Cloud-VPS (Project-requests): TfInfraTest project - https://phabricator.wikimedia.org/T365822#9829866 (10rook) ` root@cloudcontrol1005:~# openstack project create --description 'TfInfraTest' TfInfraTest --domain default +-------------+-------------+ | Field | Value | +-------------+------------... [14:51:11] 10Cloud-VPS (Project-requests): TfInfraTest project - https://phabricator.wikimedia.org/T365822#9829867 (10rook) 05Open→03Resolved a:03rook [15:00:41] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons: Adding new members to Cloud VPS project fails - https://phabricator.wikimedia.org/T365096#9829917 (10Andrew) Hello @Zache I'm unable to reproduce this issue, and I'm a bit surprised by the .mov you attached -- when I use that interface I'm unable... [15:13:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:15:39] 10Cloud-VPS: move project to TfInfraTest - https://phabricator.wikimedia.org/T365830 (10rook) 03NEW [15:16:29] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons: Adding new members to Cloud VPS project fails - https://phabricator.wikimedia.org/T365096#9829970 (10Zache) [15:16:33] (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110) [15:19:06] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.05.27 - 2024.06.16): Upgrade clouddb1021 to bookworm - https://phabricator.wikimedia.org/T365450#9830006 (10BTullis) [15:33:13] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons: Adding new members to Cloud VPS project fails - https://phabricator.wikimedia.org/T365096#9830045 (10Zache) Hi Andrew, I tested it second time at this week at it still failed so it was not random insident. I added the browser information to the t... [15:43:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:53:17] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9830093 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm [16:12:38] (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110) [16:36:27] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9830255 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm [16:45:04] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [16:54:08] 10Tool-yearinreview, 06Indic-MediaWiki-Developers, 07good first task: Shareable image download issue - https://phabricator.wikimedia.org/T364642#9830357 (10Gopavasanth) [16:57:02] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Toolforge, 07Documentation: Restructure and improve content for: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database - https://phabricator.wikimedia.org/T232404#9830374 (10fnegri) a:03fnegri [16:57:54] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9830380 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm executed with errors:... [16:58:36] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9830383 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm [17:04:13] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: [wikireplicas] Update Admin docs - https://phabricator.wikimedia.org/T365717#9830404 (10fnegri) There's also https://wikitech.wikimedia.org/wiki/Wiki_Replica_redaction which partially overlaps with https://wikitech.wikimedia.org/... [17:04:43] 10Tool-yearinreview, 06Indic-MediaWiki-Developers: Restore caching of user stats - https://phabricator.wikimedia.org/T364697#9830410 (10Gopavasanth) [17:05:04] (03CR) 10Pintoch: [C:03+1] "The `redis.conf` looks good to me. Snapshotting is disabled and I think that makes sense for a lot of applications. Thanks for working on " [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1012797 (https://phabricator.wikimedia.org/T360378) (owner: 10BryanDavis) [17:24:52] 10Tool-dabfix, 05Community-Wishlist-Survey-2023, 03Wikimedia Wishathon: Investigate Dabfix tool implementation - https://phabricator.wikimedia.org/T336545#9830499 (10Gopavasanth) [17:26:57] 10Tool-bridgebot, 07Upstream: Bridgebot freaks out and sends double messages from IRC to Telegram - https://phabricator.wikimedia.org/T305487#9830507 (10bd808) 05Stalled→03Open >>! In T305487#9750844, @bd808 wrote: > I have submitted the patch that is working in our deploy upstream: https://github.com/42wi... [17:30:21] 10Tool-dabfix, 06Indic-MediaWiki-Developers: Improvise the dabfix tool UI - https://phabricator.wikimedia.org/T365848 (10Gopavasanth) 03NEW [17:35:05] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9830591 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm [17:41:30] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9830617 (10Andrew) The main point of suspicion here is that it doesn't autoconfig the network but asks me to specify a netmask. Cathal suggests that switching the nic fir... [17:41:52] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9830621 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm executed with errors:... [17:43:04] 10Tool-toolwatch, 06Indic-MediaWiki-Developers: Sort tools based on tool Title - https://phabricator.wikimedia.org/T353579#9830618 (10Gopavasanth) a:05Saivarshini70→03None [17:46:00] 10Tool-dabfix, 06Indic-MediaWiki-Developers: Update README for Dabfix Tool - https://phabricator.wikimedia.org/T365854 (10Gopavasanth) 03NEW [17:53:42] FIRING: [3x] CloudVPSDesignateLeaks: Detected 7 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:55:50] 10Tool-toolwatch: Improve the functionality of Toolwatch - https://phabricator.wikimedia.org/T353573#9830693 (10Gopavasanth) @Aklapper, thank you! All issues related to the tool can now be tracked only at this link: https://phabricator.wikimedia.org/tag/tool-toolwatch/ [17:58:42] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 7 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:09:55] 10Tool-toolwatch, 06Indic-MediaWiki-Developers: Enhance Tool Watch to Track Tool Liveliness and Display Graphs - https://phabricator.wikimedia.org/T365857 (10Gopavasanth) 03NEW [18:10:10] (03update) 10raymond-ndibe: [jobs-cli] support services in jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/18 (https://phabricator.wikimedia.org/T348758) [18:10:25] (03update) 10raymond-ndibe: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 (https://phabricator.wikimedia.org/T348758) [18:42:42] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:47:42] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:52:42] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:57:42] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:20:27] (03update) 10raymond-ndibe: [jobs-cli] support services in jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/18 (https://phabricator.wikimedia.org/T348758) [19:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:41:00] (03update) 10raymond-ndibe: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 (https://phabricator.wikimedia.org/T348758) [20:41:07] (03update) 10raymond-ndibe: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 (https://phabricator.wikimedia.org/T348758) [20:41:26] (03update) 10raymond-ndibe: [jobs-cli] support services in jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/18 (https://phabricator.wikimedia.org/T348758) [20:42:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-32 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:45:04] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [23:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks