[00:51:00] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [00:51:01] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:51:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [00:51:12] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [00:51:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [01:21:18] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [01:21:20] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [01:21:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [01:21:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [02:23:29] 10Toolforge, 07Kubernetes: toolforge-jobs and packbuild images - https://phabricator.wikimedia.org/T369786#9999914 (10Hawkeye7) That's exactly what I did. I used envvars list to hold the password: ` tools.milhistbot@tools-bastion-13:~$ toolforge envvars list name value CREDX_MILHISTBOT_... [03:47:19] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [03:47:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [03:47:31] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [03:47:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [04:17:48] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [04:17:50] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [04:17:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [04:17:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [04:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:02:22] (03PS1) 10Majavah: Add jobs.yaml [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055515 [06:18:22] (03CR) 10Majavah: [C:03+2] Add jobs.yaml [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055515 (owner: 10Majavah) [06:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:19:42] (03Merged) 10jenkins-bot: Add jobs.yaml [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055515 (owner: 10Majavah) [06:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:36:39] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:41:39] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:44:32] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [06:44:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [06:44:44] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [06:44:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:14:25] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [07:14:27] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [07:14:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:14:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:21:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [07:26:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [07:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:37:32] (03PS1) 10Majavah: Add timeouts to scheduled jobs [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055518 [08:39:18] (03CR) 10Majavah: [C:03+2] Add timeouts to scheduled jobs [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055518 (owner: 10Majavah) [08:40:14] (03Merged) 10jenkins-bot: Add timeouts to scheduled jobs [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055518 (owner: 10Majavah) [09:39:38] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [09:39:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:39:50] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [09:39:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:43:38] (03CR) 10Majavah: cookbooks: add new wmcs.openstack.tofu cookbook (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [09:48:25] (03PS1) 10Majavah: Add logrotate config [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055519 [09:49:33] (03CR) 10Majavah: [C:03+2] Add logrotate config [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055519 (owner: 10Majavah) [09:50:31] (03Merged) 10jenkins-bot: Add logrotate config [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1055519 (owner: 10Majavah) [10:11:44] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [10:11:46] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [10:11:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:11:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:56:23] 06cloud-services-team, 10Toolforge: [infra,k8s] Move to kubernetes PAVs and drop kyverno - https://phabricator.wikimedia.org/T364293#10000111 (10taavi) [10:56:24] 10Toolforge: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#10000112 (10taavi) [10:56:25] 10Toolforge: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#10000114 (10taavi) [10:56:26] 06cloud-services-team, 10Toolforge: [infra,k8s] Move to kubernetes PAVs and drop kyverno - https://phabricator.wikimedia.org/T364293#10000113 (10taavi) [11:34:56] (03PS2) 10Majavah: tools: Setting SCM or issue tracker URL is optional [labs/striker] - 10https://gerrit.wikimedia.org/r/1052355 (https://phabricator.wikimedia.org/T369395) (owner: 10BryanDavis) [11:35:46] (03PS3) 10Majavah: tools: Setting SCM or issue tracker URL is optional [labs/striker] - 10https://gerrit.wikimedia.org/r/1052355 (https://phabricator.wikimedia.org/T369395) (owner: 10BryanDavis) [11:36:15] (03PS4) 10Majavah: tools: Setting SCM or issue tracker URL is optional [labs/striker] - 10https://gerrit.wikimedia.org/r/1052355 (https://phabricator.wikimedia.org/T369395) (owner: 10BryanDavis) [11:37:03] (03CR) 10Majavah: [C:03+2] tools: Setting SCM or issue tracker URL is optional [labs/striker] - 10https://gerrit.wikimedia.org/r/1052355 (https://phabricator.wikimedia.org/T369395) (owner: 10BryanDavis) [11:38:15] (03Merged) 10jenkins-bot: tools: Setting SCM or issue tracker URL is optional [labs/striker] - 10https://gerrit.wikimedia.org/r/1052355 (https://phabricator.wikimedia.org/T369395) (owner: 10BryanDavis) [12:32:36] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Rebuild or delete deployment-docker-mobileapps01 - https://phabricator.wikimedia.org/T369915#10000163 (10Southparkfan) @Jgiannelos I see you have created mobileapps02 with a Bullseye image. Is mobileapps01 ready f... [12:37:15] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [12:37:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:37:27] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [12:37:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:02:51] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10000187 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye [13:07:20] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [13:07:22] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [13:07:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:07:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:10:46] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10000188 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephmon1005.eqiad.wmnet with OS bullseye [13:19:57] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace or delete deployment-mediawiki[11-12].deployement-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361387#10000191 (10Southparkfan) mediawiki11 and mediawiki12 are no l... [13:26:04] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-push-notifications01.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370459#10000192 (10Southparkfan) a:03Southparkfan [13:34:30] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-push-notifications01.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370459#10000196 (10Southparkfan) ^ after merg... [13:39:00] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10000197 (10Southparkfan) a:03Southparkfan [13:43:28] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10000200 (10Southparkfan) Upgrade to bullseye/bookworm blocked due to T332015.... [14:04:46] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-parsoid12.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361386#10000233 (10Southparkfan) a:05jijiki→03Southparkfan `deployment-pa... [14:09:56] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-mwmaint02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T370582 (10Southparkfan) 03NEW [14:10:02] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-mwmaint02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T370582#10000245 (10Southparkfan) a:03Southparkfan [14:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:20:49] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10000248 (10Jclark-ctr) @Papaul 1004 will still not start to image i checked Vlans and compared to 1003 with no luck also. [14:23:00] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10000251 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye ex... [14:25:42] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#10000252 (10Southparkfan) [14:27:01] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-shellbox.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370462#10000255 (10Southparkfan) a:03Southparkfan [14:31:01] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10000257 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudcephmon1005.eqiad.wmnet with OS bullseye ex... [14:45:02] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-shellbox.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370462#10000266 (10Southparkfan) It looks like the shellbox container is broken... [14:46:35] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-shellbox.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370462#10000268 (10Southparkfan) 05Open→03Resolved [14:55:20] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-mwmaint02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T370582#10000280 (10Southparkfan) Given that Puppet does not have a flag to stop the periodic MediaWiki... [15:03:08] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370465#10000285 (10Southparkfan) @BTullis I see you have created `deployment-... [15:15:31] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-urldownloader03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370466#10000297 (10Southparkfan) a:03Southparkfan [15:17:36] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-urldownloader03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370466#10000295 (10Southparkfan) As soon as the ab... [15:23:29] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10000299 (10Southparkfan) @hnowlan I see you have created `deployment-maps-master02`. Other than pos... [15:34:10] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [15:34:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:34:21] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [15:34:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:36:30] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10000302 (10Southparkfan) @Jgiannelos hey! Is `deployment-restbase-bul... [15:37:03] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461#10000305 (10Southparkfan) a:03Southparkfan [15:52:00] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461#10000308 (10Southparkfan) Great, can't ssh into my new instance: `... [16:04:28] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [16:04:30] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [16:04:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:04:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:46:59] (03CR) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [18:19:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:29:31] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [18:29:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:29:43] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [18:29:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:59:36] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [18:59:38] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [18:59:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:59:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:26:00] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [21:26:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:26:11] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [21:26:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:56:08] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [21:56:10] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [21:56:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:56:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:19:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks