[00:16:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:21:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:22:14] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: 14Update devtools project puppetmaster - 14https://phabricator.wikimedia.org/T360470#9708900 (10Dzahn) 14I went through the other instances in this project and switched to new puppetmaster in: -... [00:33:22] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api] replace all error message models with ResponseMessages - https://phabricator.wikimedia.org/T361901#9708906 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/61 [builds-cli... [00:33:43] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api] replace all error message models with ResponseMessages - https://phabricator.wikimedia.org/T361901#9708907 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/85 [builds-api... [00:43:03] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [00:44:30] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api] replace all error message models with ResponseMessages - https://phabricator.wikimedia.org/T361901#9708909 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/63 d/changelog... [00:44:30] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] Figure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#9708910 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforg... [00:45:34] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api] replace all error message models with ResponseMessages - https://phabricator.wikimedia.org/T361901#9708911 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/63 d/changelog... [00:45:34] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] Figure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#9708912 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforg... [00:56:17] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) [00:58:35] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [01:09:51] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [01:09:52] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [01:09:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:10:52] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [01:10:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:11:03] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [01:11:03] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [01:11:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:12:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:12:48] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [01:12:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:13:41] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [01:13:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:13:48] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcontrol2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [01:14:17] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission [01:14:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:14:37] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission [01:14:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:15:11] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission [01:15:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:15:30] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission [01:15:30] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission [01:15:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:15:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:16:15] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [01:16:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:16:19] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission [01:16:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:16:25] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [01:16:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:16:51] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission [01:16:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:17:07] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [01:17:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:17:18] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [01:17:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:17:32] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico [01:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:17:39] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico [01:17:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:17:43] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission [01:17:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:18:24] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [01:18:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:18:26] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico [01:18:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:18:30] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico [01:18:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:18:39] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [01:18:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:19:18] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [01:19:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:19:36] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [01:19:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:22:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:28:52] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) [02:05:35] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [02:16:28] (PuppetAgentFailure) firing: (2) Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-27 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [02:21:18] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) [02:55:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-test-k8s-etcd-29 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [02:58:08] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [02:58:44] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [02:58:57] dependabot[bot] opened https://github.com/toolforge/paws/pull/400 [03:06:31] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [03:10:28] (PuppetAgentNoResources) resolved: No Puppet resources found on instance toolsbeta-test-k8s-etcd-29 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [03:17:31] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [03:36:28] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [03:37:01] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [03:56:28] (PuppetAgentFailure) firing: (3) Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-27 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [04:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:22:18] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [04:22:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:35:17] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [04:37:00] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [04:41:28] (PuppetAgentFailure) firing: (3) Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-27 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [04:42:28] (InstanceDown) firing: Project toolsbeta instance toolsbeta-test-k8s-etcd-29 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:46:08] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [04:46:38] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [04:47:12] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [04:47:28] (InstanceDown) resolved: Project toolsbeta instance toolsbeta-test-k8s-etcd-29 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:48:02] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [04:48:35] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [04:51:09] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [05:01:26] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [05:06:28] (PuppetAgentFailure) firing: (2) Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-27 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [05:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:12:56] (SystemdUnitDown) firing: (2) The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:17:56] (SystemdUnitDown) resolved: (2) The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:37:21] (03PS1) 10Majavah: Add jobs.yaml [labs/tools/train-blockers] - 10https://gerrit.wikimedia.org/r/1019148 [06:37:51] (03PS2) 10Majavah: Add jobs.yaml [labs/tools/train-blockers] - 10https://gerrit.wikimedia.org/r/1019148 [06:39:57] (03PS3) 10Majavah: Add jobs.yaml [labs/tools/train-blockers] - 10https://gerrit.wikimedia.org/r/1019148 [06:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:45:01] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Set up a bitu instance for codfw1dev - https://phabricator.wikimedia.org/T360795#9709110 (10MoritzMuehlenhoff) >>! In T360795#9707776, @Andrew wrote: > I'd prefer that it go on its own ganeti VM, as I'm trying to pare down on the total number if weird... [06:45:46] (03CR) 10Majavah: [V:03+2 C:03+2] Add jobs.yaml [labs/tools/train-blockers] - 10https://gerrit.wikimedia.org/r/1019148 (owner: 10Majavah) [06:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:57:55] 10Toolforge: [component-api] Develop the webhook mechanism to trigger a deployment - https://phabricator.wikimedia.org/T362066#9709116 (10Slst2020) [07:54:05] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [api-gateway] Add a python server to serve consolidated openapi docs - https://phabricator.wikimedia.org/T362299#9709185 (10CodeReviewBot) sstefanova opened https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/17 Draft: WIP:... [08:09:54] (03CR) 10Majavah: [C:03+2] labsauth: Update UI labels to use 'developer account' term [labs/striker] - 10https://gerrit.wikimedia.org/r/1018942 (owner: 10Majavah) [08:12:17] (03Merged) 10jenkins-bot: labsauth: Update UI labels to use 'developer account' term [labs/striker] - 10https://gerrit.wikimedia.org/r/1018942 (owner: 10Majavah) [08:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:22:35] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent - https://phabricator.wikimedia.org/T362233#9709252 (10dcaro) [08:28:33] 10Toolforge: [jobs-cli,jobs-api] Provide a means to configure a task to be restarted indefinately upon error, but terminate normally otherwise - https://phabricator.wikimedia.org/T361405#9709262 (10aborrero) [08:32:29] 10Toolforge: 14Java jobs run the Stretch grid seem to require a very large memory reservation - 14https://phabricator.wikimedia.org/T219351#9709267 (10aborrero) 05Open→03Declined 14We no longer have the Grid Engine backend in Toolforge. [08:38:25] 06cloud-services-team, 10Striker, 13Patch-For-Review: 14Deploy Striker instance for toolsbeta - 14https://phabricator.wikimedia.org/T360025#9709274 (10taavi) 05Open→03Resolved [08:41:23] 10Striker, 13Patch-For-Review: 14Put "Dev Environment Warning" on striker.wmflabs.org - 14https://phabricator.wikimedia.org/T254598#9709276 (10taavi) 05Open→03Resolved a:03taavi 14There is now support for configuring a banner, and the demo server is currently down (T329687). [08:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:09:06] 06cloud-services-team, 10Cloud-VPS: 14Expired cert failure on cloudinfra-cloudvps-puppetserver-1.cloudinfra.eqiad1.wikimedia.cloud - 14https://phabricator.wikimedia.org/T361772#9709303 (10taavi) 05Open→03Resolved 14`lang=irc 18:56:18 taavi: can you advise about the last comment on T3617... [09:13:10] 10Cloud-VPS, 10DNS, 06SRE, 06Traffic: 14DNS name resolution failure with www.spacecom.mil from Cloud VPS - 14https://phabricator.wikimedia.org/T346471#9709332 (10taavi) 05Open→03Resolved a:03taavi [09:15:33] 10Toolforge: [lima-kilo] toolforge_deploy_mr.py confusing message - https://phabricator.wikimedia.org/T362389 (10Slst2020) 03NEW [09:23:36] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [09:23:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:24:10] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [09:24:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:27:15] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [09:27:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:27:58] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [09:28:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:30:32] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison [09:30:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:30:55] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison [09:30:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:31:23] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission [09:31:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:31:52] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission [09:31:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:34:42] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission [09:34:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:35:15] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission [09:35:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:43:48] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [cicd,infra] pre-cache all the pre-commit hooks - https://phabricator.wikimedia.org/T362314#9709402 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/32 precommit: add code to open MRs to update pre... [10:09:21] 10Toolforge: [docs] update READMEs - https://phabricator.wikimedia.org/T362390 (10Slst2020) 03NEW [10:10:49] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission [10:10:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:11:19] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission [10:11:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:14:17] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission [10:14:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:14:48] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission [10:14:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:41:09] 10Toolforge: [infra,builds-builder] "failed to create fsnotify watcher: too many open files" - https://phabricator.wikimedia.org/T361519#9709522 (10dcaro) not 100% sure if it's the same, but found this issue also when trying to follow a pod log from a control node: ` root@tools-k8s-control-9:~# kubectl -n envva... [10:52:32] 10Toolforge: [infra,builds-builder] "failed to create fsnotify watcher: too many open files" - https://phabricator.wikimedia.org/T361519#9709549 (10dcaro) Inside the container, the limit is also big, I think this might be related to one of k8s helper processes: ` root@tools-k8s-worker-102:~# crictl exec -ti 4b82... [10:54:22] 10Toolforge: [infra,builds-builder] "failed to create fsnotify watcher: too many open files" - https://phabricator.wikimedia.org/T361519#9709553 (10dcaro) it's continuously trying to run a pod, from the `teg` tool: ` root@tools-k8s-control-9:~# kubectl get all -n tool-teg NAME READY... [12:17:02] 10Toolforge: [infra,builds-builder] "failed to create fsnotify watcher: too many open files" - https://phabricator.wikimedia.org/T361519#9709707 (10dcaro) Yep, it's a system-wide issue: ` root@tools-k8s-worker-102:~# tail -f firstboot_done tail: inotify cannot be used, reverting to polling: Too many open files... [12:17:23] 06cloud-services-team, 10wikitech.wikimedia.org, 10Observability-Alerting: Move wikitech-static monitoring off Icinga - https://phabricator.wikimedia.org/T362397 (10taavi) 03NEW [12:20:59] 10Toolforge: [infra,builds-builder] "failed to create fsnotify watcher: too many open files" - https://phabricator.wikimedia.org/T361519#9709724 (10dcaro) Yep, manually changing the value sorts out the issue \o/ [12:21:33] (03PS1) 10Elukey: role::cassandra_dev: add fake truststore password for PKI [labs/private] - 10https://gerrit.wikimedia.org/r/1019274 (https://phabricator.wikimedia.org/T352647) [12:36:32] 10Toolforge, 13Patch-For-Review: [infra,builds-builder] "failed to create fsnotify watcher: too many open files" - https://phabricator.wikimedia.org/T361519#9709770 (10dcaro) Give it a couple hours to run puppet on all the workers, but after that, @bd808 can you notify if you see any more in the next few days? [12:37:12] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [infra,builds-builder] "failed to create fsnotify watcher: too many open files" - https://phabricator.wikimedia.org/T361519#9709774 (10dcaro) a:03dcaro [12:37:42] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [infra,builds-builder] "failed to create fsnotify watcher: too many open files" - https://phabricator.wikimedia.org/T361519#9709776 (10dcaro) [12:37:49] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [infra,builds-builder] "failed to create fsnotify watcher: too many open files" - https://phabricator.wikimedia.org/T361519#9709779 (10dcaro) 05Open→03In progress [12:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:55:15] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [api-gateway] Add a python server to serve consolidated openapi docs - https://phabricator.wikimedia.org/T362299#9709816 (10Slst2020) [13:15:47] 14Toolforge Build Service: Rust buildservice fails - https://phabricator.wikimedia.org/T362404 (10Magnus) 03NEW [13:20:58] 10Cloud-VPS, 07SecTeam-Processed, 07Security, 07Vuln-Infoleak: 14Improper Access Control on timeless.wmflabs.org - 14https://phabricator.wikimedia.org/T239730#9709944 (10sbassett) [13:21:36] 14Toolforge Build Service: Rust buildservice fails - https://phabricator.wikimedia.org/T362404#9709949 (10dcaro) This looks like github timing out: ` fatal: unable to access 'https://github.com/magnusmanske/mixnmatch_rs/': Failed to connect to github.com port 443 after 130492 ms: Operation timed out ` does it h... [13:22:13] 10Cloud-VPS, 07SecTeam-Processed, 07Security, 07Vuln-Infoleak: 14Improper Access Control on timeless.wmflabs.org - 14https://phabricator.wikimedia.org/T239730#9709942 (10sbassett) a:03taavi [13:23:39] 14Toolforge Build Service: Rust buildservice fails - https://phabricator.wikimedia.org/T362404#9709950 (10Magnus) It did happen three times in a row before I reported it, just worked now. Maybe a github problem or a server not seeing internet? [13:29:25] 14Toolforge Build Service: Rust buildservice fails - https://phabricator.wikimedia.org/T362404#9709968 (10dcaro) It did pull the images for the container (might have happened before though) and seemed to be able to resolve github.com (the dns is not local afaik) though of course, things might be cached/not all c... [13:34:46] 14Toolforge Build Service: Rust buildservice fails - https://phabricator.wikimedia.org/T362404#9709983 (10dcaro) No connectivity issues to github found: ` root@cloudcumin1001:~# cumin 'O{project:tools}' 'nc -w 1 -vz github.com 443' 101 hosts will be targeted: tools-acme-chief-[3-4].tools.eqiad1.wikimedia.cloud,t... [13:52:28] 10Toolforge (Toolforge iteration 08): [cicd,infra] pre-cache all the pre-commit hooks - https://phabricator.wikimedia.org/T362314#9710037 (10dcaro) Hmm, I'm not 100% convinced of the current approach, it will still need to download stuff when running golangci-lint (~1G for builds-api): ` dcaro@urcuchillay$ podm... [14:00:25] 10Toolforge (Toolforge iteration 08): [cicd,infra] pre-cache all the pre-commit hooks - https://phabricator.wikimedia.org/T362314#9710072 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/31 py3.11-bookworm-tox: add pre-commit caching [14:00:41] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [14:04:14] 10Toolforge (Toolforge iteration 08): [cicd,infra] pre-cache all the pre-commit hooks - https://phabricator.wikimedia.org/T362314#9710081 (10dcaro) The first run was way faster than expected (<2min): https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/240230 Nice, let's see how it works during... [14:08:44] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [14:09:19] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [14:17:12] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [14:21:28] (PuppetAgentFailure) resolved: Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-27 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [14:34:08] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [14:39:42] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [14:40:04] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [14:45:43] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [14:45:53] 10Toolforge: Rust buildservice failed to clone a repository from GitHub - https://phabricator.wikimedia.org/T362404#9710241 (10taavi) [14:45:53] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Set up a bitu instance for codfw1dev - https://phabricator.wikimedia.org/T360795#9710239 (10Andrew) > > Fair enough. Do you have any preference on the name? cloudidm2001.codfw.wmnet, e.g.? idm2001-dev.codfw.wmnet (or, possibly, cloudidm2001-dev.codfw... [14:46:34] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [14:59:58] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) [15:05:30] (03CR) 10Eevans: [C:03+1] role::cassandra_dev: add fake truststore password for PKI [labs/private] - 10https://gerrit.wikimedia.org/r/1019274 (https://phabricator.wikimedia.org/T352647) (owner: 10Elukey) [15:08:56] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [15:11:35] (03CR) 10Elukey: [V:03+2 C:03+2] role::cassandra_dev: add fake truststore password for PKI [labs/private] - 10https://gerrit.wikimedia.org/r/1019274 (https://phabricator.wikimedia.org/T352647) (owner: 10Elukey) [15:14:24] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [15:15:54] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [15:21:28] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [15:32:19] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node [15:45:45] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) [16:03:35] 06cloud-services-team, 10Toolforge, 07Kubernetes, 13Patch-For-Review: 14[infra] Upgrade Toolforge K8s etcd nodes to Bullseye - 14https://phabricator.wikimedia.org/T349207#9710465 (10Andrew) 05Open→03Resolved a:03Andrew 14etcd nodes are now all bullseye. Moving them to bookworm will require more... [16:12:02] 06cloud-services-team, 10Toolforge, 07Kubernetes, 13Patch-For-Review: 14[infra] Upgrade Toolforge K8s etcd nodes to Bullseye - 14https://phabricator.wikimedia.org/T349207#9710487 (10taavi) 14>>! In T349207#9710465, @Andrew wrote: > etcd nodes are now all bullseye. Moving them to bookworm will require... [16:23:41] 10PAWS: Remove prometheus migrate logic - https://phabricator.wikimedia.org/T362102#9710591 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/401 [16:23:56] vivian-rook opened https://github.com/toolforge/paws/pull/401 [17:09:34] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9710710 (10Dzahn) jenkins-releases.devtools was deleted by jnuche after he confirmed it was once his test instance and not used anymore [17:10:06] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9710711 (10Dzahn) [17:17:14] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install cloudcontrol2006-dev.codfw.wmnet - https://phabricator.wikimedia.org/T354896#9710739 (10Jhancock.wm) [17:39:11] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:46:47] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: 14Update devtools project puppetmaster - 14https://phabricator.wikimedia.org/T360470#9710780 (10Dzahn) 14One issue: on deploy-1004 (which itself is on buster!) we are getting "Failed to generate... [18:12:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:22:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:32:37] 06cloud-services-team, 10decommission-hardware: decommission cloudbackup200[12].codfw.wmnet - https://phabricator.wikimedia.org/T362438 (10Andrew) 03NEW [18:46:17] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudbackup200[12].codfw.wmnet - https://phabricator.wikimedia.org/T362438#9710837 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudbackup2001.codfw.wmnet` - cloudbackup2001.c... [18:56:31] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudbackup200[12].codfw.wmnet - https://phabricator.wikimedia.org/T362438#9710845 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudbackup2002.codfw.wmnet` - cloudbackup2002.c... [18:58:26] 06cloud-services-team, 10decommission-hardware, 10ops-codfw, 13Patch-For-Review: decommission cloudbackup200[12].codfw.wmnet - https://phabricator.wikimedia.org/T362438#9710846 (10Andrew) [19:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:11:26] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: 14Update devtools project puppetmaster - 14https://phabricator.wikimedia.org/T360470#9711015 (10Andrew) 14It's not really a buster thing -- the puppet code for geoip is entirely different in the... [21:34:26] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Learn how to do what Taavi does - https://phabricator.wikimedia.org/T362443 (10Andrew) 03NEW [21:34:28] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9711038 (10Dzahn) 05Resolved→03Open [21:36:02] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9711043 (10bd808) test [21:42:57] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: maintain-kubeusers - https://phabricator.wikimedia.org/T362444 (10Andrew) 03NEW [21:43:34] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: Toolforge k8s upgrades - https://phabricator.wikimedia.org/T362445 (10Andrew) 03NEW [21:44:11] 06cloud-services-team, 10Cloud-VPS: 14Request to add catalyst-qte.wmcloud.org webproxy subdomain for the catalyst-qte CloudVPS project - 14https://phabricator.wikimedia.org/T361517#9711067 (10EBomani) 14Thank you so much for this! It works and we were able to close out the ticket.  [21:45:05] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: toolforge job investigation - https://phabricator.wikimedia.org/T362446 (10Andrew) 03NEW [21:48:21] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: Toolforge misc services (e.g. mail server) - https://phabricator.wikimedia.org/T362447 (10Andrew) 03NEW [21:49:23] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: rebuild toolforge docker images - https://phabricator.wikimedia.org/T362448 (10Andrew) 03NEW [21:51:12] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: python-flask-keystone - https://phabricator.wikimedia.org/T362449 (10Andrew) 03NEW [21:52:30] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: Cloud VPS OpenTofu provider - https://phabricator.wikimedia.org/T362450 (10Andrew) 03NEW [21:53:39] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: cloud-vps monitoring - https://phabricator.wikimedia.org/T362452 (10Andrew) 03NEW [22:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:36:13] 10Toolforge: 14[envvars-cli] Either hide or show envvars values, but not both - 14https://phabricator.wikimedia.org/T359558#9711232 (10EBomani) 05Open→03Resolved [23:43:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:53:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks