[00:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:48:48] 06cloud-services-team, 10Toolforge: toolforge jobs load flushes out all jobs - https://phabricator.wikimedia.org/T364204#10388838 (10Raymond_Ndibe) please @Multichill share your jobs.yaml file so I can attempt reproducing this and see exactly what is happening [08:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:02:48] 06cloud-services-team, 10Toolforge: toolforge jobs load errors with 404 repetatively - https://phabricator.wikimedia.org/T381273#10388844 (10Urbanecm) >>! In T381273#10371500, @Slst2020 wrote: > Has this worked for you in the past, then started failing today? Yes, I definitely loaded the jobs via that command... [09:09:12] 06cloud-services-team, 10Toolforge: toolforge jobs load errors with 404 repetatively - https://phabricator.wikimedia.org/T381273#10388855 (10taavi) Here's the stack trace from the jobs-api side: ` Traceback (most recent call last): File "/opt/lib/poetry/tjf-9TtSrW0h-py3.11/lib/python3.11/site-packages/flask/... [11:21:13] 06cloud-services-team, 10Toolforge: toolforge jobs load flushes out all jobs - https://phabricator.wikimedia.org/T364204#10388892 (10Raymond_Ndibe) I investigated this a bit. I think the problem is coming from the replica field. For some reason I forgot to account for that in loads since it was added after the... [13:12:20] (03open) 10raymond-ndibe: [toolforge-deploy] refetch default to main branch [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/645 (https://phabricator.wikimedia.org/T358225) [13:12:29] (03update) 10raymond-ndibe: [toolforge-deploy] refetch default to main branch [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/645 (https://phabricator.wikimedia.org/T358225) [13:13:01] (03open) 10raymond-ndibe: [jobs-api] replicas default to 1 in NewJob model [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T364204) [13:14:04] (03update) 10raymond-ndibe: [jobs-api] replicas default to 1 in NewJob model [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T364204) [13:58:33] (03open) 10raymond-ndibe: [lima-kilo] deploy maintain-harbor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/217 (https://phabricator.wikimedia.org/T358225) [13:58:47] (03update) 10raymond-ndibe: [lima-kilo] deploy maintain-harbor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/217 (https://phabricator.wikimedia.org/T358225) [13:59:00] (03update) 10raymond-ndibe: [lima-kilo] deploy maintain-harbor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/217 (https://phabricator.wikimedia.org/T358225) [15:53:10] (03approved) 10raymond-ndibe: [lima-kilo] deploy maintain-harbor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/217 (https://phabricator.wikimedia.org/T358225) [15:53:14] (03merge) 10raymond-ndibe: [lima-kilo] deploy maintain-harbor [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/217 (https://phabricator.wikimedia.org/T358225) [16:16:22] (03open) 10raymond-ndibe: [toolforge-deploy] add more test cases to job loads [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/646 (https://phabricator.wikimedia.org/T364204) [16:33:43] (03update) 10raymond-ndibe: [toolforge-deploy] add more test cases to job loads [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/646 (https://phabricator.wikimedia.org/T364204) [17:14:49] (03update) 10raymond-ndibe: [jobs-api] replicas default to 1 in NewJob model [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T364204) [17:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks