[01:46:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:57:26] (03open) 10raymond-ndibe: [jobs-cli] remove to_delete from calculate_changes [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T364204) [01:57:43] (03update) 10raymond-ndibe: [jobs-cli] remove to_delete from calculate_changes [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T364204) [02:02:14] (03update) 10raymond-ndibe: [jobs-cli] remove to_delete from calculate_changes [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T364204) [02:07:47] (03update) 10raymond-ndibe: [jobs-cli] remove to_delete from calculate_changes [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T364204) [02:08:53] 10Toolforge (Toolforge iteration 10): [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#9836357 (10Raymond_Ndibe) [02:12:58] (03update) 10raymond-ndibe: [jobs-cli] remove to_delete from calculate_changes [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T364204) [02:19:32] (03update) 10raymond-ndibe: [jobs-cli] remove to_delete from calculate_changes [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T364204) [03:06:49] FIRING: DiskSpace: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 5.054% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:46:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:06:49] FIRING: DiskSpace: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 5.054% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:12:31] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [07:16:30] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [07:18:44] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [07:20:08] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [07:31:00] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [08:05:15] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [08:05:39] 10Toolforge, 06Project-Admins: Create project tag for toolforge_i18n - https://phabricator.wikimedia.org/T365939#9836667 (10LucasWerkmeister) [08:44:19] (03update) 10aborrero: deployment: introduce restartPolicy: never [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/26 [08:44:43] (03close) 10aborrero: deployment: introduce restartPolicy: never [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/26 [08:53:01] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [08:59:32] (03open) 10dcaro: Draft: functional_tests: add script to run them [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/133 [09:14:43] 10VPS-project-Wikistats: Add dtpwiki to wikistats - https://phabricator.wikimedia.org/T365234#9836853 (10RhinosF1) 05Stalled→03In progress a:03RhinosF1 [09:19:21] 10Data-Services, 06DBA: Prepare and check storage layer for dtpwiki - https://phabricator.wikimedia.org/T365229#9836875 (10Zabe) Wiki has been created [09:20:55] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [09:25:49] 10VPS-project-Wikistats: Add dtpwiki to wikistats - https://phabricator.wikimedia.org/T365234#9836896 (10RhinosF1) 05In progress→03Resolved Wiki Created by @zabe ` MariaDB [wikistats]> insert into wikipedias (prefix,lang,loclang,method) values ("dtp", "Central Dusun", "Kadazandusun", 8); Query OK, 1 ro... [09:35:43] (03update) 10dcaro: Draft: functional_tests: add script to run them [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/133 [09:38:52] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [09:44:29] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 [09:46:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:51:49] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [10:04:34] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [10:08:12] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [10:08:31] (03update) 10dcaro: Draft: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [10:08:49] (03update) 10dcaro: Draft: functional_tests: add script to run them [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/133 [10:10:08] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [10:10:13] (03update) 10dcaro: functional_tests: add script to run them [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/133 [10:17:29] (03update) 10dcaro: functional_tests: add script to run them [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/133 [10:20:57] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [10:31:43] (03open) 10dcaro: jobs-api: listen on 127.0.0.1 by default [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/86 [10:33:59] (03approved) 10aborrero: jobs-api: listen on 127.0.0.1 by default [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/86 (owner: 10dcaro) [10:37:27] (03update) 10dcaro: jobs-api: listen on 127.0.0.1 by default [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/86 [10:37:31] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [10:40:56] (03approved) 10dcaro: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 (https://phabricator.wikimedia.org/T348758) (owner: 10raymond-ndibe) [10:40:58] (03unapproved) 10dcaro: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 (https://phabricator.wikimedia.org/T348758) (owner: 10raymond-ndibe) [10:40:59] (03approved) 10dcaro: jobs-api: listen on 127.0.0.1 by default [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/86 [10:41:05] (03update) 10dcaro: jobs-api: listen on 127.0.0.1 by default [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/86 [10:41:13] (03merge) 10dcaro: jobs-api: listen on 127.0.0.1 by default [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/86 [10:43:24] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [10:43:26] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.302-20240528104119-80e50d7b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/289 [10:43:34] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [10:43:36] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [10:44:17] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [10:44:29] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [10:45:06] (03approved) 10dcaro: jobs-api: bump to 0.0.302-20240528104119-80e50d7b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/289 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:45:08] (03update) 10dcaro: jobs-api: bump to 0.0.302-20240528104119-80e50d7b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/289 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:46:57] (03approved) 10dcaro: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 (https://phabricator.wikimedia.org/T348758) (owner: 10raymond-ndibe) [10:46:58] (03update) 10dcaro: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 (https://phabricator.wikimedia.org/T348758) (owner: 10raymond-ndibe) [10:47:12] (03update) 10dcaro: [jobs-cli] support services in jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/18 (https://phabricator.wikimedia.org/T348758) (owner: 10raymond-ndibe) [10:47:13] (03approved) 10dcaro: [jobs-cli] support services in jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/18 (https://phabricator.wikimedia.org/T348758) (owner: 10raymond-ndibe) [10:51:01] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [11:06:49] FIRING: DiskSpace: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 5.054% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:07:14] 10PAWS: Update images to 24.04 - https://phabricator.wikimedia.org/T366055 (10rook) 03NEW [12:08:17] 10PAWS: nbserve to 24.04 - https://phabricator.wikimedia.org/T366056 (10rook) 03NEW [12:08:41] 10PAWS: update renderer to 24.04 - https://phabricator.wikimedia.org/T366057 (10rook) 03NEW [12:09:16] 10PAWS: update singleuser to 24.04 - https://phabricator.wikimedia.org/T366058 (10rook) 03NEW [12:37:05] 06cloud-services-team, 10Toolforge (Toolforge iteration 10), 13Patch-For-Review: [maintain-kubeusers,infra,k8s]: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) - https://phabricator.wikimedia.org/T364312#9837513 (10aborrero) apparently, the error is related to s... [12:46:57] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [12:49:22] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:49:33] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:51:25] 06cloud-services-team: toolforge: package k9s for use in kubernetes - https://phabricator.wikimedia.org/T366061 (10aborrero) 03NEW [12:54:13] 10PAWS: minesweeper image to alpine 3.20.0 - https://phabricator.wikimedia.org/T366063 (10rook) 03NEW [12:54:55] 10PAWS: jobber image to alpine 3.20.0 - https://phabricator.wikimedia.org/T366064 (10rook) 03NEW [12:57:05] 10PAWS: Update images - https://phabricator.wikimedia.org/T366055#9837593 (10rook) [12:59:00] vivian-rook opened https://github.com/toolforge/paws/pull/416 [12:59:40] 06cloud-services-team, 10Toolforge (Toolforge iteration 10), 13Patch-For-Review: [maintain-kubeusers,infra,k8s]: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) - https://phabricator.wikimedia.org/T364312#9837647 (10aborrero) now deployed in toolsbeta without errors [13:01:01] 06cloud-services-team: toolforge: package k9s for use in kubernetes - https://phabricator.wikimedia.org/T366061#9837673 (10aborrero) upstream publishes .deb files in their release page: https://github.com/derailed/k9s/releases [13:16:35] RESOLVED: DiskSpace: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 5.054% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:17:15] 06cloud-services-team: cloud: puppetmasters: adopt cinder volumes to store certs and git repos - https://phabricator.wikimedia.org/T276327#9837754 (10Andrew) 05Open→03Resolved a:03Andrew I rebuilt all the puppetservers (for puppet 7) last month and moved everything onto a cinder volume. The docs at ht... [13:19:44] 06cloud-services-team, 10Cloud-VPS: 'backy2 cleanup' not getting called properly on cloudbackup hosts - https://phabricator.wikimedia.org/T366071 (10Andrew) 03NEW [13:24:34] 06cloud-services-team, 10Cloud-VPS: cloudinfra-db backups - https://phabricator.wikimedia.org/T301640#9837804 (10Andrew) I wouldn't say it's a proper solution, but we are now backing up up both VMs and cinder volumes wholesale with backy. That makes this less urgent, but saving raw mysql dumps would still be... [13:46:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:51:42] FIRING: [3x] CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:56:42] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:06:34] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:07:42] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [14:16:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:17:20] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [14:41:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:43:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:49:54] 10PAWS: update singleuser to 24.04 - https://phabricator.wikimedia.org/T366058#9838119 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/417 [14:50:02] vivian-rook opened https://github.com/toolforge/paws/pull/417 [14:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:52:04] 10VPS-project-Wikistats: Add dtpwiki to wikistats - https://phabricator.wikimedia.org/T365234#9838150 (10Dzahn) Thank you :) [14:53:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:16:05] 10Toolforge (Toolforge iteration 10): [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#9838239 (10dcaro) [15:16:34] 10Toolforge (Toolforge iteration 10): [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#9838244 (10dcaro) >>! In T359650#9836271, @Raymond_Ndibe wrote: > made some attempt to define somethings and answer some important questions on the task description, based on our discussion @... [15:20:17] 10Toolforge: [infra] Allow users to self-rotate the all the credentials - https://phabricator.wikimedia.org/T365724#9838259 (10dcaro) p:05Triage→03Medium [15:21:07] 06cloud-services-team, 10Toolforge: [infra,k8s] package k9s for use in kubernetes - https://phabricator.wikimedia.org/T366061#9838252 (10dcaro) p:05Triage→03Low [15:37:55] 10PAWS: update renderer to 24.04 - https://phabricator.wikimedia.org/T366057#9838322 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/418 [15:38:01] vivian-rook opened https://github.com/toolforge/paws/pull/418 [15:41:23] FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM [15:46:23] RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM [16:22:58] 10PAWS: Investigate adding quarto - https://phabricator.wikimedia.org/T366090 (10rook) 03NEW [16:41:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:50:07] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [16:51:29] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [16:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:58:37] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [17:02:04] 10PAWS: update renderer to 24.04 - https://phabricator.wikimedia.org/T366057#9838937 (10github-toolforge-bot) andrewbogott closed https://github.com/toolforge/paws/pull/418 [17:02:14] andrewbogott closed https://github.com/toolforge/paws/pull/418 [17:05:20] (03approved) 10dcaro: [jobs-cli] remove to_delete from calculate_changes [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T364204) (owner: 10raymond-ndibe) [17:05:22] (03update) 10dcaro: [jobs-cli] remove to_delete from calculate_changes [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T364204) (owner: 10raymond-ndibe) [17:19:45] 10PAWS: minesweeper image to alpine 3.20.0 - https://phabricator.wikimedia.org/T366063#9839001 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/416 [17:19:48] vivian-rook closed https://github.com/toolforge/paws/pull/416 [17:20:01] 10PAWS: minesweeper image to alpine 3.20.0 - https://phabricator.wikimedia.org/T366063#9839002 (10rook) 05Open→03Resolved a:03rook [17:20:18] 10PAWS: update renderer to 24.04 - https://phabricator.wikimedia.org/T366057#9839008 (10rook) 05Open→03Resolved a:03Andrew [17:30:33] 10PAWS: nbserve to 24.04 - https://phabricator.wikimedia.org/T366056#9839068 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/419 [17:30:50] vivian-rook opened https://github.com/toolforge/paws/pull/419 [17:37:39] 10PAWS: nbserve to 24.04 - https://phabricator.wikimedia.org/T366056#9839116 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/419 [17:37:48] vivian-rook closed https://github.com/toolforge/paws/pull/419 [17:38:22] 10PAWS: nbserve to 24.04 - https://phabricator.wikimedia.org/T366056#9839117 (10rook) 05Open→03Resolved a:03rook [17:40:11] 10PAWS: jobber image to alpine 3.20.0 - https://phabricator.wikimedia.org/T366064#9839128 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/420 [17:40:19] vivian-rook opened https://github.com/toolforge/paws/pull/420 [17:40:39] FIRING: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:41:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 7 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:45:39] RESOLVED: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 7 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:15:08] 10PAWS: jobber image to alpine 3.20.0 - https://phabricator.wikimedia.org/T366064#9839276 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/420 [18:15:19] vivian-rook closed https://github.com/toolforge/paws/pull/420 [18:16:35] 10PAWS: jobber image to alpine 3.20.0 - https://phabricator.wikimedia.org/T366064#9839284 (10rook) 05Open→03Resolved a:03rook [18:19:37] 10PAWS: Investigate adding quarto - https://phabricator.wikimedia.org/T366090#9839301 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/421 [18:19:44] vivian-rook opened https://github.com/toolforge/paws/pull/421 [18:31:47] 14Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9839361 (10MBH) @dcaro A script `sandbox.py` generates an error `Skipped '/workspace/user-config.py': owned by someone else.` I don't see `workspace` folder on T... [18:39:50] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, and 2 others: Degraded RAID on cloudcephosd1031 - https://phabricator.wikimedia.org/T364060#9839429 (10Jclark-ctr) 05Open→03Resolved Replaced Failed Drive [18:47:31] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [18:48:41] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [18:57:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [18:57:56] 10Quarry: move testing off blubberoid - https://phabricator.wikimedia.org/T366107 (10rook) 03NEW [18:59:02] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [19:00:46] 10PAWS: Investigate adding quarto - https://phabricator.wikimedia.org/T366090#9839565 (10rook) Seems to work. Generated: https://public-paws.wmcloud.org/User:VRook_(WMF)/quarto.html with `quarto render quarto.ipynb --to html` [19:01:06] 10PAWS: Investigate adding quarto - https://phabricator.wikimedia.org/T366090#9839568 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/421 [19:01:18] vivian-rook closed https://github.com/toolforge/paws/pull/421 [19:01:48] 10PAWS: Investigate adding quarto - https://phabricator.wikimedia.org/T366090#9839570 (10rook) 05Open→03Resolved a:03rook [19:03:56] 10PAWS: tox - https://phabricator.wikimedia.org/T366109 (10rook) 03NEW [19:04:29] 10PAWS: helm-lint image - https://phabricator.wikimedia.org/T366110 (10rook) 03NEW [19:05:54] 10PAWS: helm-lint image - https://phabricator.wikimedia.org/T366110#9839617 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/422 [19:06:08] vivian-rook opened https://github.com/toolforge/paws/pull/422 [19:10:12] 10PAWS: helm-lint image - https://phabricator.wikimedia.org/T366110#9839624 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/422 [19:10:17] 10PAWS: helm-lint image - https://phabricator.wikimedia.org/T366110#9839625 (10rook) 05Open→03Resolved a:03rook [19:10:19] vivian-rook closed https://github.com/toolforge/paws/pull/422 [19:11:13] 10PAWS: tox - https://phabricator.wikimedia.org/T366109#9839629 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/423 [19:11:25] vivian-rook opened https://github.com/toolforge/paws/pull/423 [19:14:45] 10PAWS: tox - https://phabricator.wikimedia.org/T366109#9839637 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/423 [19:14:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [19:14:56] vivian-rook closed https://github.com/toolforge/paws/pull/423 [19:15:19] 10PAWS: tox - https://phabricator.wikimedia.org/T366109#9839634 (10rook) 05Open→03Resolved a:03rook [19:16:02] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [19:17:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [19:21:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [19:21:17] 10Quarry: tox - https://phabricator.wikimedia.org/T366112 (10rook) 03NEW [19:21:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [19:22:15] 10Quarry: tox - https://phabricator.wikimedia.org/T366112#9839688 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/44 [19:22:26] vivian-rook opened https://github.com/toolforge/quarry/pull/44 [19:22:35] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [19:24:08] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [19:24:30] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [19:27:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [19:28:14] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [19:31:41] 10Tool-yearinreview: Please attribute the original, add disclaimer and add the LICENSE - https://phabricator.wikimedia.org/T366114 (10Jdlrobson) 03NEW [19:32:12] 10Quarry: move testing off blubberoid - https://phabricator.wikimedia.org/T366107#9839736 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/45 [19:32:20] vivian-rook opened https://github.com/toolforge/quarry/pull/45 [19:34:27] 10Quarry: move testing off blubberoid - https://phabricator.wikimedia.org/T366107#9839743 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/45 [19:34:44] vivian-rook closed https://github.com/toolforge/quarry/pull/45 [19:36:22] 10Tool-yearinreview, 06Indic-MediaWiki-Developers, 07good first task: Shareable image download issue - https://phabricator.wikimedia.org/T364642#9839741 (10Jdlrobson) > Going forward, we will maintain our code repository exclusively on GitLab: Year in Review GitLab Repository. Clarification: As I explained... [19:37:29] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [19:38:12] 10Quarry: tox - https://phabricator.wikimedia.org/T366112#9839746 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/44 [19:38:20] vivian-rook closed https://github.com/toolforge/quarry/pull/44 [19:38:41] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [19:39:51] 10Quarry: move testing off blubberoid - https://phabricator.wikimedia.org/T366107#9839748 (10rook) 05Open→03Resolved [19:39:55] 10Quarry: tox - https://phabricator.wikimedia.org/T366112#9839749 (10rook) 05Open→03Resolved [19:43:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [19:52:44] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [19:53:02] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [19:59:47] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [20:15:08] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9839817 (10GTrang) 05Open→03Resolved a:03GTrang Bug appears to have been fixed. [20:39:52] 10PAWS: Update chart version on PR? - https://phabricator.wikimedia.org/T365725#9839916 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/424 [20:40:00] vivian-rook opened https://github.com/toolforge/paws/pull/424 [20:52:30] 10PAWS: Update chart version on PR? - https://phabricator.wikimedia.org/T365725#9839954 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/424 [20:52:37] vivian-rook closed https://github.com/toolforge/paws/pull/424 [21:01:40] vivian-rook opened https://github.com/toolforge/paws/pull/425 [21:01:44] 10PAWS: Update chart version on PR? - https://phabricator.wikimedia.org/T365725#9840057 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/425 [21:23:21] vivian-rook opened https://github.com/toolforge/paws/pull/426 [21:27:30] 10PAWS: ingress-nginx not idempotent - https://phabricator.wikimedia.org/T366121 (10rook) 03NEW [21:28:29] 10PAWS: ingress-nginx not idempotent - https://phabricator.wikimedia.org/T366121#9840157 (10rook) [21:29:36] 10PAWS: prometheus not idempotent - https://phabricator.wikimedia.org/T366122 (10rook) 03NEW [21:30:07] vivian-rook closed https://github.com/toolforge/paws/pull/426 [21:33:59] vivian-rook opened https://github.com/toolforge/paws/pull/427 [21:36:42] vivian-rook closed https://github.com/toolforge/paws/pull/427 [21:46:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 14 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:53:26] 10Cloud-VPS, 10PAWS: periodic error from tofu when state in object storage - https://phabricator.wikimedia.org/T366124 (10rook) 03NEW [22:28:08] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9840338 (10Andrew) The gitpuppet patch was missing on three other puppet servers: taavi-puppet7-server.testlabs.eqiad1.wikimed... [22:35:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-app10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:35:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:36:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:37:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tf-bastion on project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:37:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance etcd-discovery-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:40:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:40:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:40:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:45:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:47:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:50:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:50:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:52:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tf-bastion on project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:52:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:55:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:55:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:55:28] RESOLVED: [3x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:56:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:56:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-nfs-1 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:57:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:00:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:01:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:02:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:05:28] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:07:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:11:28] RESOLVED: [3x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:12:28] RESOLVED: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources