[00:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:47:58] 10Tool-meetbot, 10Internet-Archive, 10WMF-General-or-Unknown, 07OKR-Work: Let MeetBot logs be indexed and archived - https://phabricator.wikimedia.org/T58533#10132446 (10Pppery) [04:39:25] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1071597 (owner: 10L10n-bot) [05:17:37] (03Abandoned) 10Pppery: Archive repo [labs/tools/dawiki] - 10https://gerrit.wikimedia.org/r/1047146 (https://phabricator.wikimedia.org/T270105) (owner: 10Pppery) [06:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:51:10] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Data-Persistence, 06Data-Persistence-SRE: [wikireplicas] Update Admin docs - https://phabricator.wikimedia.org/T365717#10132681 (10ABran-WMF) >>! In T365717#10130071, @fnegri wrote: > @ABran-WMF thank you for reviewing! > >> I've found that par... [08:13:32] 10cloud-services-team (FY2024/2025-Q1-Q2): Drain C8 rack - https://phabricator.wikimedia.org/T374043#10132718 (10dcaro) [08:15:35] 10cloud-services-team (FY2024/2025-Q1-Q2): Drain C8 rack - https://phabricator.wikimedia.org/T374043#10132732 (10dcaro) [08:15:36] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, and 2 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544#10132731 (10dcaro) [08:22:57] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [ceph,network] Intermittent network packets lost - https://phabricator.wikimedia.org/T371869#10132744 (10dcaro) 05Open→03Resolved a:03dcaro This was sorted out by hard-rebooting the switch that was misbehaving, and has not happened again (someth... [08:23:09] 10cloud-services-team (FY2024/2025-Q1-Q2): Drain C8 rack - https://phabricator.wikimedia.org/T374043#10132763 (10dcaro) [08:24:38] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, and 2 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544#10132754 (10dcaro) @cmooney @VRiley-WMF Hi! I'm almost done draining the rack, we can try to find a slot startin... [08:27:53] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, and 2 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544#10132768 (10dcaro) [08:45:29] !log dcaro@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (T373986) [08:45:35] T373986: cloudsw1-c8-eqiad is unstable - https://phabricator.wikimedia.org/T373986 [08:46:36] !log dcaro@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T373986) [08:54:12] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [08:54:15] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [08:54:27] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [08:57:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [08:57:53] (03approved) 10sstefanova: ChecksDashboard: add scheduled job check [toolforge-repos/sample-complex-app-frontend] (fix_color_scheme) - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/6 (owner: 10dcaro) [08:58:15] (03approved) 10sstefanova: cronjob: add simple cronjob [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/3 (https://phabricator.wikimedia.org/T368602) (owner: 10dcaro) [08:59:50] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [08:59:52] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:00:04] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:03:05] (03PS1) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: print tofu validate errors [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1071813 [09:03:27] (03approved) 10sstefanova: show task status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370321) (owner: 10dcaro) [09:03:32] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:03:36] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:03:43] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:06:48] (03approved) 10sstefanova: Add deployment steps info [toolforge-repos/sample-complex-app-frontend] (show_task_status) - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/3 (https://phabricator.wikimedia.org/T372478) (owner: 10dcaro) [09:06:53] (03PS2) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: print tofu validate errors [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1071813 [09:07:02] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:07:06] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:07:26] (03PS3) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: print tofu validate errors [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1071813 [09:07:28] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:07:35] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:08:22] (03update) 10dcaro: cronjob: add simple cronjob [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/3 (https://phabricator.wikimedia.org/T368602) [09:08:56] (03approved) 10sstefanova: checks: add database check [toolforge-repos/sample-complex-app-frontend] (show_deployment_steps) - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/4 (https://phabricator.wikimedia.org/T370317) (owner: 10dcaro) [09:09:14] (03PS4) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: print tofu init and validate errors [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1071813 [09:10:21] (03approved) 10sstefanova: Fix color schemes for non-dark mode [toolforge-repos/sample-complex-app-frontend] (add_database_check) - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/5 (owner: 10dcaro) [09:10:47] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] wmcs.openstack.tofu: print tofu init and validate errors [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1071813 (owner: 10Arturo Borrero Gonzalez) [09:12:57] (03merge) 10dcaro: show task status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370321) [09:12:57] (03update) 10dcaro: Add deployment steps info [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/3 (https://phabricator.wikimedia.org/T372478) [09:14:35] (03Merged) 10jenkins-bot: wmcs.openstack.tofu: print tofu init and validate errors [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1071813 (owner: 10Arturo Borrero Gonzalez) [09:16:59] (03update) 10dcaro: Add deployment steps info [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/3 (https://phabricator.wikimedia.org/T372478) [09:17:14] (03merge) 10dcaro: Add deployment steps info [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/3 (https://phabricator.wikimedia.org/T372478) [09:17:14] (03update) 10dcaro: checks: add database check [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/4 (https://phabricator.wikimedia.org/T370317) [09:17:24] (03update) 10dcaro: checks: add database check [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/4 (https://phabricator.wikimedia.org/T370317) [09:17:35] (03merge) 10dcaro: checks: add database check [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/4 (https://phabricator.wikimedia.org/T370317) [09:17:35] (03update) 10dcaro: Fix color schemes for non-dark mode [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/5 [09:17:45] (03update) 10dcaro: Fix color schemes for non-dark mode [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/5 [09:17:51] (03merge) 10dcaro: Fix color schemes for non-dark mode [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/5 [09:17:52] (03update) 10dcaro: ChecksDashboard: add scheduled job check [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/6 [09:18:01] (03update) 10dcaro: ChecksDashboard: add scheduled job check [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/6 [09:18:15] (03merge) 10dcaro: ChecksDashboard: add scheduled job check [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/6 [09:19:02] (03merge) 10dcaro: cronjob: add simple cronjob [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/3 (https://phabricator.wikimedia.org/T368602) [09:19:41] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:20:00] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:20:00] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:20:35] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:21:53] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14): [sct.frontend] show the current deployment stats - https://phabricator.wikimedia.org/T372478#10132917 (10dcaro) 05In progress→03Resolved [09:22:02] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 07Epic, 13Patch-For-Review: [Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment - https://phabricator.wikimedia.org/T368602#10132913 (10dcaro) 05In progre... [09:22:17] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:22:44] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:27:44] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10132930 (10dcaro) a:05dcaro→03Raymond_Ndibe [09:27:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudlb2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:31:36] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:32:41] !log dcaro@urcuchillay cloudinfra-codfw1dev START - Cookbook wmcs.openstack.cloudvirt.vm_console [09:32:42] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [09:32:48] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudlb1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:32:49] !log dcaro@urcuchillay cloudinfra-codfw1dev END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [09:32:49] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [09:32:57] !log dcaro@urcuchillay cloudinfra-codfw1dev START - Cookbook wmcs.openstack.cloudvirt.vm_console [09:32:57] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [09:33:02] !log dcaro@urcuchillay cloudinfra-codfw1dev END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [09:33:03] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [09:33:13] !log dcaro@urcuchillay cloudinfra-codfw1dev START - Cookbook wmcs.openstack.cloudvirt.vm_console [09:33:14] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [09:33:19] !log dcaro@urcuchillay cloudinfra-codfw1dev END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [09:33:19] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [09:33:24] !log dcaro@urcuchillay cloudinfra-codfw1dev START - Cookbook wmcs.openstack.cloudvirt.vm_console [09:33:24] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [09:33:33] !log dcaro@urcuchillay cloudinfra-codfw1dev END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1) [09:33:33] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [09:36:08] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:42:48] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on cloudlb1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:46:22] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:47:48] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudlb1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:48:30] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:48:45] FIRING: WidespreadPuppetFailure: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [09:52:12] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:52:26] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:52:55] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:59:24] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [09:59:26] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [09:59:34] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [10:00:14] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [10:00:18] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [10:00:51] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [10:03:29] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [10:04:26] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [10:15:30] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [10:22:48] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudlb1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:27:48] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudlb1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:28:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [10:32:48] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudlb1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:37:48] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cloudlb1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:47:48] RESOLVED: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudlb1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:04:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:24:11] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [11:26:43] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [11:29:03] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [11:36:19] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [11:36:23] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [11:37:02] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [11:46:25] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [12:02:42] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [12:02:42] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:03:13] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:05:18] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:05:20] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [12:05:51] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:05:57] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:06:27] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:07:32] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [12:07:50] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:08:06] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:14:05] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:14:09] (03update) 10aborrero: tofu-infra: introduce DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 (https://phabricator.wikimedia.org/T374338) [12:14:18] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:15:11] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:15:22] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40 [12:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:31:09] !log dcaro@urcuchillay cloudinfra-codfw1dev START - Cookbook wmcs.openstack.cloudvirt.vm_console [12:31:09] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [13:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:01:11] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:02:11] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:16:31] 10Tool-meetbot, 10Internet-Archive, 10WMF-General-or-Unknown, 07OKR-Work: Let MeetBot logs be indexed and archived - https://phabricator.wikimedia.org/T58533#10133851 (10hashar) 05Open→03Resolved We might had a Meetbot instance originally setup in the `integration` project, that is why the link ref... [13:31:09] (03merge) 10dcaro: jobs,cronjobs: add clarifying note on why the limits [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/60 (https://phabricator.wikimedia.org/T372720) [13:35:01] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.168-20240910133124-0c3e395c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/519 (https://phabricator.wikimedia.org/T372720) [13:57:39] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster (T359641) [13:57:43] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [13:57:43] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:04:05] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster [14:04:06] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:11:52] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster (T359641) [14:11:56] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:13:10] 06cloud-services-team: Update wmcloud.org MX records - https://phabricator.wikimedia.org/T374278#10134202 (10aborrero) I was hoping to finish the work to support DNS records on our opentofu setup, see {T374338} because otherwise this value is hardcoded in the openstack database. [14:21:02] !log dcaro@cloudcumin1001 toolsbeta Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster [14:21:02] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster [14:25:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster (T359641) [14:25:53] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:35:49] !log dcaro@cloudcumin1001 toolsbeta Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster [14:35:50] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster [14:36:14] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster (T359641) [14:36:16] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:46:47] !log dcaro@cloudcumin1001 toolsbeta Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster [14:46:47] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster [14:47:25] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster (T359641) [14:47:26] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:47:26] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [14:57:10] !log raymond-ndibe@cloudcumin1001 toolsbeta Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster [14:57:10] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster [14:57:11] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:57:11] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [14:57:18] 10cloud-services-team (FY2024/2025-Q1-Q2): [cloud] Drain B row from cloud* services - https://phabricator.wikimedia.org/T374463 (10dcaro) 03NEW [14:58:19] 10cloud-services-team (FY2024/2025-Q1-Q2): [cloud] Drain B row from cloud* services - https://phabricator.wikimedia.org/T374463#10134485 (10dcaro) [15:05:34] 10cloud-services-team (FY2024/2025-Q1-Q2): [cloud] Drain B row from cloud* services - https://phabricator.wikimedia.org/T374463#10134519 (10dcaro) [15:15:00] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-1 (T359641) [15:15:02] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:15:02] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:15:47] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-1 (T359641) [15:15:47] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:22:22] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-2 (T359641) [15:22:23] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:22:23] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:22:35] 10cloud-services-team (FY2024/2025-Q1-Q2): 2024-09-10: hardware error on cloudvirt2004-dev - https://phabricator.wikimedia.org/T374467 (10dcaro) 03NEW p:05Triage→03Medium [15:23:09] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-2 (T359641) [15:23:09] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:24:48] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-3 (T359641) [15:24:48] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:25:55] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-3 (T359641) [15:25:55] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:27:14] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-4 (T359641) [15:27:14] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:28:01] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-4 (T359641) [15:28:02] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [15:28:03] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [15:41:18] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: tofu-infra: extend coverage to Designate DNS data - https://phabricator.wikimedia.org/T374338#10134701 (10aborrero) found a potential bug in the upstream provider for openstack, sent a patch here: https://github.com/terraform-provider-openstack/terrafor... [15:56:50] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10134779 (10Raymond_Ndibe) [16:02:51] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: MVP: Privately serve wikitech via mwdebug1001 - https://phabricator.wikimedia.org/T371537#10134805 (10jijiki) [16:07:39] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: MVP: Privately serve wikitech via mwdebug1001 - https://phabricator.wikimedia.org/T371537#10134797 (10jijiki) 05In progress→03Resolved Further testing completed with @Ladsgroup * [[https://logstash.wikimedia.org/goto/45728dfc... [16:11:22] !log dcaro@urcuchillay catalyst START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:11:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [16:13:22] FIRING: MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [16:15:30] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-test-k8s-worker-nfs-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:30:15] !log dcaro@urcuchillay catalyst END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:30:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [16:58:29] 10Cloud-VPS (Quota-requests): Add 160Gb storage - https://phabricator.wikimedia.org/T374476 (10SDunlap) 03NEW [17:02:30] 10Cloud-VPS (Quota-requests): Add 160Gb storage - https://phabricator.wikimedia.org/T374476#10135063 (10Slst2020) +1 [17:03:16] 10Cloud-VPS (Quota-requests): Add 160Gb storage - https://phabricator.wikimedia.org/T374476#10135067 (10dcaro) 05Open→03In progress a:03dcaro [17:03:32] !log dcaro@urcuchillay catalyst START - Cookbook wmcs.openstack.quota_increase (T374476) [17:03:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [17:03:36] T374476: Add 160Gb storage - https://phabricator.wikimedia.org/T374476 [17:03:42] !log dcaro@urcuchillay catalyst END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T374476) [17:03:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Catalyst/SAL [17:06:50] 10Cloud-VPS (Quota-requests): Add 160Gb storage - https://phabricator.wikimedia.org/T374476#10135085 (10dcaro) 05In progress→03Resolved Done :) [17:14:08] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster (T359641) [17:14:10] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:14:10] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [17:14:52] RESOLVED: MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [17:23:15] !log raymond-ndibe@cloudcumin1001 toolsbeta Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster [17:23:16] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster [17:23:17] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:23:17] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:49:36] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-6 (T359641) [17:49:38] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:49:38] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [17:54:50] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-6 [17:54:51] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:55:27] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-6 (T359641) [17:55:27] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [17:55:28] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [18:00:34] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-6 [18:00:35] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [18:03:58] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-6 (T359641) [18:03:58] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [18:03:58] T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641 [18:06:37] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=97) for host toolsbeta-test-k8s-worker-nfs-6 [18:06:38] logmsgbot_cloud: Unknown project "raymond-ndibe@cloudcumin1001" [18:19:24] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-worker-nfs-6 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [18:20:30] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-test-k8s-worker-nfs-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:45:13] !log dcaro@urcuchillay cloudinfra-codfw1dev END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [18:45:15] wmbot~dcaro@urcuchillay: Unknown project "cloudinfra-codfw1dev" [19:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:34:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks