[00:09:49] <wmcs-alerts>	 FIRING: TfInfraTestApplyFailed: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed
[00:56:03] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Epic, 07IPv6: Enable IPv6 on CloudVPS - https://phabricator.wikimedia.org/T37947#9849070 (10Dzahn) Just wanted to add this is still a source of trouble for us when trying to use cloud VPS to have test machines.  We keep having to work around it a lot in different conte...
[01:12:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:17:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:22:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:27:41] <jinxer-wm>	 RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:41:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:46:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:56:41] <jinxer-wm>	 RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:09:45] <wikibugs>	 (03CR) 10Legoktm: [C:03+2] "Sorry about the delay in reviewing! Thanks :)" [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/1034212 (https://phabricator.wikimedia.org/T365416) (owner: 10Brian Wolff)
[04:10:47] <wikibugs>	 (03Merged) 10jenkins-bot: Ensure file name uses 7 character rev even if ambigious [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/1034212 (https://phabricator.wikimedia.org/T365416) (owner: 10Brian Wolff)
[07:43:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[07:53:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[08:59:27] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9849621 (10fnegri) Thanks @Andrew. I think we can probably resolve this as your patch should fix most of the issues. We can reo...
[09:41:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:53:19] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110)
[09:56:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:59:47] <wikibugs>	 (03update) 10aborrero: helpers: add foxtrot-ldap-load-users.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/129
[10:14:14] <wikibugs>	 (03open) 10aborrero: functional-tests: increase retry default time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/292
[10:18:26] <wikibugs>	 (03approved) 10ladsgroup: bacc: use `<code/>` rather than `<pre/>` in alert [toolforge-repos/deploy-commands] - 10https://gitlab.wikimedia.org/toolforge-repos/deploy-commands/-/merge_requests/3 (owner: 10bd808)
[10:18:28] <wikibugs>	 (03merge) 10ladsgroup: bacc: use `<code/>` rather than `<pre/>` in alert [toolforge-repos/deploy-commands] - 10https://gitlab.wikimedia.org/toolforge-repos/deploy-commands/-/merge_requests/3 (owner: 10bd808)
[10:20:37] <wikibugs>	 (03update) 10aborrero: functional-tests: increase retry default time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/292
[11:23:22] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110)
[11:28:50] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110)
[11:49:48] <wikibugs>	 10VPS-project-Extdist, 10ExtensionDistributor, 13Patch-For-Review: Hash collision on ExtDist, occurs 404 Not Found on Extension Distributor - https://phabricator.wikimedia.org/T340882#9849865 (10Bawolff) 05Open→03Resolved a:03Bawolff An alternate fix was merged (Still 7 chars, but use 7 even if amb...
[12:26:41] <jinxer-wm>	 RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[12:37:47] <wikibugs>	 10PAWS: update singleuser to 24.04 - https://phabricator.wikimedia.org/T366058#9849962 (10rook) Rstudio is installing, but not R notebook/console
[12:38:32] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312)
[12:44:41] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312)
[13:00:14] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110)
[13:39:54] <wikibugs>	 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9850052 (10Don-vip)
[13:42:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:52:41] <jinxer-wm>	 RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:10:05] <wikibugs>	 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons, 13Patch-For-Review: Adding new (nonexistent) members to Cloud VPS project fails - https://phabricator.wikimedia.org/T365096#9850126 (10Andrew)
[14:10:34] <wikibugs>	 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons, 13Patch-For-Review: Adding new (nonexistent) members to Cloud VPS project fails without useful feedback - https://phabricator.wikimedia.org/T365096#9850128 (10Andrew)
[14:12:47] <wikibugs>	 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons, 13Patch-For-Review: Adding new (nonexistent) members to Cloud VPS project fails without useful feedback - https://phabricator.wikimedia.org/T365096#9850134 (10Andrew) 05Open→03Resolved Thanks for the followup @Zache .  I've re-titled this...
[14:31:49] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110)
[14:36:21] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312)
[14:36:35] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Puppet (Puppet 7.0): cloud-vps puppetservers filling up / with puppetserver reports - https://phabricator.wikimedia.org/T366357 (10Andrew) 03NEW
[14:37:04] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110)
[15:06:33] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 10), 13Patch-For-Review: [maintain-kubeusers,infra,k8s]: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) - https://phabricator.wikimedia.org/T364312#9850259 (10aborrero) I think https://gitlab.wikimedia.org/...
[15:31:38] <wikibugs>	 (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110)
[15:46:26] <notefromgithub>	 andrewbogott opened https://github.com/toolforge/paws/pull/431
[15:46:43] <wikibugs>	 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T366226#9850478 (10github-toolforge-bot) andrewbogott opened https://github.com/toolforge/paws/pull/431
[15:58:38] <wikibugs>	 10Toolforge: [toolforge] [redis] Improve Puppet config - https://phabricator.wikimedia.org/T366365#9850563 (10fnegri)
[15:59:04] <wikibugs>	 10Toolforge: [toolforge] [redis] Improve Puppet config - https://phabricator.wikimedia.org/T366365#9850566 (10fnegri)
[16:01:56] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: wmcs_libs: add image controller to refactor upload function [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1037821
[16:01:56] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: wmcs.toolforge.k8s: add cookbook to upload kyverno container images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1037822 (https://phabricator.wikimedia.org/T364113)
[16:06:46] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 10): [toolforge] Redis refusing connections - https://phabricator.wikimedia.org/T363709#9850593 (10fnegri) > (still unresolved) I expected the patch to add a line timeout 600 to /etc/redis/tcp_6379.conf, but for some reason it did not...
[16:16:55] <wikibugs>	 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T366226#9850663 (10github-toolforge-bot) andrewbogott closed https://github.com/toolforge/paws/pull/431
[16:17:05] <notefromgithub>	 andrewbogott closed https://github.com/toolforge/paws/pull/431
[16:25:36] <wikibugs>	 10Data-Services: [wikireplicas] clouddb* free memory decreases over time - https://phabricator.wikimedia.org/T365164#9850701 (10fnegri) New memory alerts started triggering for clouddb1016 and clouddb1020. I will restart them on Monday.
[16:34:55] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[35-38] - https://phabricator.wikimedia.org/T363344#9850746 (10VRiley-WMF) cloudcephosd1035 Rack: C8 U 28 CableID: 5335 Port: 20   cloudcephosd1036 Rack: D5 U 18 CableID: 5337 Port: 18  cloudcephosd1037 Rack...
[16:41:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[17:22:38] <wikibugs>	 10Cloud-VPS (Quota-requests): Increase Object Storage quota for QRank - https://phabricator.wikimedia.org/T366244#9850992 (10bd808) +1
[17:22:54] <wikibugs>	 10Cloud-VPS (Quota-requests): Increase Object Storage quota for QRank - https://phabricator.wikimedia.org/T366244#9850994 (10rook) +1
[17:28:57] <wikibugs>	 10Cloud-VPS (Quota-requests): Increase Object Storage quota for QRank - https://phabricator.wikimedia.org/T366244#9851005 (10Andrew) 05Open→03Resolved a:03Andrew I've adjusted these quotas -- please re-open if I missed anything!
[18:36:39] <wikibugs>	 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9851225 (10Don-vip)
[18:41:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[18:45:03] <wikibugs>	 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9851270 (10Don-vip)
[18:51:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[18:56:41] <jinxer-wm>	 RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[19:16:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-10 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[19:44:04] <wikibugs>	 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T366226#9851469 (10rook) 05Open→03Resolved
[19:56:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-10 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[20:02:15] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Set up a bitu instance for codfw1dev - https://phabricator.wikimedia.org/T360795#9851492 (10Andrew) a:05Andrew→03SLyngshede-WMF
[20:02:16] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: 'backy2 cleanup' not getting called properly on cloudbackup hosts - https://phabricator.wikimedia.org/T366071#9851488 (10Andrew) 05Open→03Resolved I just now manually ran 'backy2 cleanup' and there was nothing to clean. So the above patch seems...
[20:02:35] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Gather feedback from users of the 'unmanaged' debian-12.0-nopuppet image - https://phabricator.wikimedia.org/T355963#9851500 (10Andrew) 05Open→03Resolved
[20:09:35] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9851552 (10Andrew) 05In progress→03Resolved
[20:11:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[20:21:28] <wmcs-alerts>	 FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on cloudinfra-internal-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[20:26:28] <wmcs-alerts>	 FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[20:46:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:23:22] <jinxer-wm>	 FIRING: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[21:28:22] <jinxer-wm>	 FIRING: [3x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[21:33:05] <wikibugs>	 06cloud-services-team: Document and enforce restrictions on openstack project names - https://phabricator.wikimedia.org/T366301#9851773 (10Andrew) 05Open→03Resolved Keystone already enforces that new project names be url safe, which is part of the battle here. With the closing of T343158, CamelCase shoul...
[21:33:22] <jinxer-wm>	 RESOLVED: [3x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[21:33:52] <wikibugs>	 06cloud-services-team: Support OpenStack projects where project name != project id - https://phabricator.wikimedia.org/T343158#9851781 (10Andrew) 05Open→03Resolved This is resolved for future projects. Existing hyphenated-project-names will remain forever broken.
[21:34:00] <wikibugs>	 06cloud-services-team: radosgw+keystone chokes on projects with '-' in their id - https://phabricator.wikimedia.org/T341509#9851785 (10Andrew) 05Open→03Resolved This is as fixed as it's going to get.
[21:39:49] <wikibugs>	 (03PS1) 10BCornwall: Move ncmonitor credentials to its own profile [labs/private] - 10https://gerrit.wikimedia.org/r/1037857
[21:46:28] <wmcs-alerts>	 RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on project-proxy-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[21:56:28] <wmcs-alerts>	 RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on cloudinfra-internal-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[22:21:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 22 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[22:26:41] <jinxer-wm>	 RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 22 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[23:13:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[23:23:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks