[00:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:39:28] (03PS1) 10Pppery: Archive repo [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/1040858 (https://phabricator.wikimedia.org/T269703) [00:39:42] (03CR) 10CI reject: [V:04-1] Archive repo [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/1040858 (https://phabricator.wikimedia.org/T269703) (owner: 10Pppery) [00:42:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:52:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:10:23] FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM [02:12:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:15:23] RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM [02:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:22:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:07:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:12:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:22:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:42:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:52:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:22:21] 10Toolforge, 06Language-Team: https://cxdebugger.toolforge.org/ has become very slow - https://phabricator.wikimedia.org/T367022 (10Nikerabbit) 03NEW [07:30:11] (03update) 10sstefanova: Draft: dev: add pre-commit [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/37 [07:31:40] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/37 [07:41:40] 10Tool-refill: refill continuous integration is broken - https://phabricator.wikimedia.org/T367026 (10Novem_Linguae) 03NEW [07:42:56] 10Cloud-VPS, 10Data-Engineering-Operations, 06DC-Ops, 07Documentation: Request: documentation of Wikimedia Analytics Cluster and Cloud VPS environmental costs - https://phabricator.wikimedia.org/T367027 (10awight) 03NEW [07:43:35] 10Cloud-VPS, 10Data-Engineering-Operations, 06DC-Ops, 07Documentation: Request: documentation of Wikimedia Analytics Cluster and Cloud VPS environmental costs - https://phabricator.wikimedia.org/T367027#9874381 (10awight) [08:04:57] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/37 [08:06:52] (03update) 10sstefanova: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [08:08:24] 10Toolforge (Toolforge iteration 11): [envvars-api, envvars-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363809#9874402 (10Slst2020) [08:09:03] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [builds-api, builds-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363808#9874404 (10Slst2020) [08:10:31] (03approved) 10aborrero: dev: add pre-commit [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/37 (owner: 10sstefanova) [08:14:13] 10Cloud-VPS, 10Data-Engineering-Operations, 06DC-Ops, 07Documentation: Request: documentation of Wikimedia Analytics Cluster and Cloud VPS environmental costs - https://phabricator.wikimedia.org/T367027#9874410 (10Novem_Linguae) [08:15:51] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/37 [08:16:43] (03merge) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/37 [08:31:46] 10Cloud-VPS, 06Infrastructure-Foundations, 10Puppet-Core: systemd-timer-mail-wrapper should not send mail as root@wikimedia.org from Cloud VPS - https://phabricator.wikimedia.org/T367028 (10taavi) 03NEW [08:37:37] 10Cloud-VPS, 10Data-Engineering-Operations, 06DC-Ops, 07Documentation: Request: documentation of Wikimedia Analytics Cluster and Cloud VPS environmental costs - https://phabricator.wikimedia.org/T367027#9874455 (10awight) [08:37:37] (03update) 10dcaro: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [08:44:53] (03update) 10sstefanova: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [08:45:12] (03update) 10sstefanova: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [08:45:12] (03update) 10dcaro: Draft: investigate authentication [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/22 (https://phabricator.wikimedia.org/T363983) [09:07:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:16:44] (03approved) 10fnegri: Manage new flavors [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/1 (https://phabricator.wikimedia.org/T364458) (owner: 10taavi) [09:17:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:17:52] (03update) 10dcaro: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 (owner: 10aborrero) [09:18:16] (03update) 10dcaro: utils: update deb build and bump setup [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366674) (owner: 10sstefanova) [09:19:43] 10Tool-schedule-deployment: Leave a comment on the Gerrit change when it is scheduled for a backport - https://phabricator.wikimedia.org/T366763#9874551 (10kostajh) >>! In T366763#9868428, @bd808 wrote: > Is there a way to write a URL + text in a Gerrit comment so that it renders as linked text and not the liter... [09:22:17] 10Toolforge: [infra,ci] Alert when toolforge-deploy changes are not deployed - https://phabricator.wikimedia.org/T358908#9874554 (10aborrero) [09:28:48] (03update) 10aborrero: components: add kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/238 (https://phabricator.wikimedia.org/T279110) [09:34:01] (03approved) 10dcaro: components: add kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/238 (https://phabricator.wikimedia.org/T279110) (owner: 10aborrero) [09:34:06] (03update) 10dcaro: components: add kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/238 (https://phabricator.wikimedia.org/T279110) (owner: 10aborrero) [09:35:35] (03update) 10dcaro: utils: update deb build and bump setup [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/36 (https://phabricator.wikimedia.org/T366674) (owner: 10sstefanova) [09:40:46] (03update) 10sstefanova: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (owner: 10raymond-ndibe) [09:45:10] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno [09:45:32] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno [09:52:57] (03merge) 10taavi: Manage new flavors [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/1 (https://phabricator.wikimedia.org/T364458) [10:17:35] (03update) 10dcaro: utils: update deb build and bump setup [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/36 (https://phabricator.wikimedia.org/T366674) (owner: 10sstefanova) [10:19:18] (03merge) 10aborrero: components: add kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/238 (https://phabricator.wikimedia.org/T279110) [10:19:41] (03approved) 10dcaro: utils: update deb build and bump setup [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366674) (owner: 10sstefanova) [10:19:43] (03update) 10dcaro: utils: update deb build and bump setup [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366674) (owner: 10sstefanova) [10:21:18] (03open) 10taavi: cloudvps_flavors: Support setting per-project flavors per deployment [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/2 (https://phabricator.wikimedia.org/T364458) [10:23:10] 06cloud-services-team, 10Toolforge: toolforge: track PSP migration plan - https://phabricator.wikimedia.org/T364297#9874732 (10aborrero) [10:34:34] (03update) 10sstefanova: utils: update deb build and bump setup [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/36 (https://phabricator.wikimedia.org/T366674) [10:36:04] (03update) 10dcaro: utils: update deb build and bump setup [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/36 (https://phabricator.wikimedia.org/T366674) (owner: 10sstefanova) [10:36:13] (03update) 10dcaro: utils: fix code deletion [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/74 (owner: 10sstefanova) [10:36:24] (03update) 10sstefanova: utils: update deb build and bump setup [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/36 (https://phabricator.wikimedia.org/T366674) [10:38:37] (03update) 10dcaro: utils: fix code deletion [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/74 (owner: 10sstefanova) [10:38:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:38:52] (03update) 10dcaro: cli: centralize context management [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/44 (owner: 10sstefanova) [10:38:57] (03update) 10dcaro: cli: centralize context management [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/44 (owner: 10sstefanova) [10:47:00] (03open) 10aborrero: functional-tests: builds-api: don't fail to cleanup [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/321 [10:47:28] (03approved) 10dcaro: functional-tests: builds-api: don't fail to cleanup [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/321 (owner: 10aborrero) [10:47:30] (03update) 10dcaro: functional-tests: builds-api: don't fail to cleanup [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/321 (owner: 10aborrero) [10:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:48:43] (03merge) 10aborrero: functional-tests: builds-api: don't fail to cleanup [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/321 [10:51:02] (03open) 10aborrero: k8s: deploy kyverno [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/140 [11:16:13] 10Cloud-VPS (Quota-requests): mediawiki2latex disk space - https://phabricator.wikimedia.org/T367007#9874861 (10taavi) Hi. I don't [[ https://openstack-browser.toolforge.org/project/ | see a project ]] named `mediawiki2latex` - could you double check that the project name is correct (and you haven't confused it... [11:19:22] 10Cloud-VPS (Quota-requests): mediawiki2latex disk space - https://phabricator.wikimedia.org/T367007#9874864 (10dhunniger) Yes you are right mediawiki2latex.wmflabs.org is the instance name the project name collection-alt-renderer [11:28:36] (03CR) 10Ladsgroup: "Overall looks good, I'm slightly worried about getting into a trap of infinite loop. If there is some defence written, that'd be nice. The" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1039977 (https://phabricator.wikimedia.org/T366878) (owner: 10Brouberol) [11:33:36] (03update) 10sstefanova: utils: update deb build and bump setup [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366674) [11:33:46] (03merge) 10sstefanova: utils: update deb build and bump setup [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366674) [11:46:35] (03update) 10sstefanova: utils: fix code deletion [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/74 [11:47:17] (03update) 10sstefanova: utils: fix code deletion [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/74 [11:50:29] (03update) 10sstefanova: utils: fix code deletion [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/74 [11:50:37] (03update) 10sstefanova: utils: fix code deletion [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/74 [11:51:12] (03update) 10sstefanova: cli: centralize context management [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/44 [11:51:16] (03approved) 10dcaro: utils: fix code deletion [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/74 (owner: 10sstefanova) [11:51:17] (03update) 10dcaro: utils: fix code deletion [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/74 (owner: 10sstefanova) [11:51:47] (03update) 10aborrero: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 [11:54:31] (03merge) 10sstefanova: utils: fix code deletion [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/74 [12:03:39] (03update) 10aborrero: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 [12:07:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:17:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:19:34] (03open) 10sstefanova: Draft: api: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/jobs-api] (add_messages_to_all_responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/92 [12:19:59] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/11 [12:21:28] (03update) 10aborrero: k8s: deploy kyverno [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/140 [12:21:39] (03approved) 10dcaro: k8s: deploy kyverno [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/140 (owner: 10aborrero) [12:22:18] (03PS4) 10Brouberol: gitlab: iterate over all pages of results to index missing repositories [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1039977 (https://phabricator.wikimedia.org/T366878) [12:22:19] (03CR) 10Brouberol: "CI is now fixed (we were downloading the latest `types-requests` package which introduced incompatible transient dependencies with our old" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1039977 (https://phabricator.wikimedia.org/T366878) (owner: 10Brouberol) [12:22:23] (03merge) 10aborrero: k8s: deploy kyverno [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/140 [12:31:26] (03update) 10aborrero: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 [12:34:33] (03update) 10dcaro: Draft: investigate authentication [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/22 (https://phabricator.wikimedia.org/T363983) [12:34:38] (03update) 10aborrero: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 [12:36:18] (03update) 10aborrero: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 [12:46:07] (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110) [12:55:10] (03update) 10aborrero: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 [13:03:03] 06cloud-services-team, 10Toolforge: toolforge: track PSP migration plan - https://phabricator.wikimedia.org/T364297#9875245 (10aborrero) >>! In T364297#9871364, @aborrero wrote: > on the latest tests I've conducted, I detected that the kyverno mutation rule somehow clashes with the PSP mutations. > > * kyvern... [13:14:59] (03update) 10sstefanova: Draft: api: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/jobs-api] (add_messages_to_all_responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/92 [13:16:49] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9875285 (10fnegri) The permissions look similar on tools-puppetserver-01: ` fnegri@tools-puppetserver-01:~$ ls -la /srv/git/op... [13:17:40] (03approved) 10dcaro: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 (owner: 10aborrero) [13:17:41] (03update) 10dcaro: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 (owner: 10aborrero) [13:24:59] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9875311 (10fnegri) Cherry-picking is also working: ` fnegri@tools-puppetserver-01:~$ sudo -i -u gitpuppet $ git fetch https://... [13:28:46] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9875323 (10taavi) Seems like someone has manually been using the `root` user ([[ https://wikitech.wikimedia.org/wiki/Help:Proje... [13:29:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.962% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:30:46] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9875330 (10taavi) I just deleted root's gitconfig file to restore the "wrong user" warning. I don't see a git flag to disable t... [13:41:11] (03update) 10taavi: cloudvps_flavors: Support setting per-project flavors per deployment [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/2 (https://phabricator.wikimedia.org/T364458) [13:41:32] (03update) 10taavi: cloudvps_flavors: Support setting per-project flavors per deployment [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/2 (https://phabricator.wikimedia.org/T364458) [13:43:58] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9875389 (10fnegri) Maybe we could prevent the root user from running git with something like `alias git="echo run git with the... [13:44:37] (03approved) 10fnegri: cloudvps_flavors: Support setting per-project flavors per deployment [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/2 (https://phabricator.wikimedia.org/T364458) (owner: 10taavi) [13:45:27] (03update) 10taavi: cloudvps_flavors: Support setting per-project flavors per deployment [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/2 (https://phabricator.wikimedia.org/T364458) [13:45:51] (03merge) 10taavi: cloudvps_flavors: Support setting per-project flavors per deployment [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/2 (https://phabricator.wikimedia.org/T364458) [13:48:36] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9875397 (10dcaro) >>! In T358496#9868398, @Andrew wrote: > I've just learned from Taavi that app credentials don't actually provide adequ... [13:54:11] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9875429 (10taavi) Do aliases/bashrc apply when running just `sudo git foo` (without `-i`)? [14:05:12] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9875477 (10fnegri) > Do aliases/bashrc apply when running just sudo git foo (without -i)? No, good point :/ [14:08:24] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Create new g4 flavors to support hypervisor migration from Linuxbridge to OVS Neutron agents - https://phabricator.wikimedia.org/T364458#9875485 (10taavi) Updated existing flavors in codfw1dev: `lang=shell-session taavi@cloudcontrol2... [14:09:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.949% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:10:06] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9875492 (10Andrew) If you would like to experiment with tool-specific keystone auth, I've set up an example account. The creds for the tool user are in cloudcontrol1006:/... [14:17:47] 10Data-Services: [wikireplicas] clouddb* free memory decreases over time - https://phabricator.wikimedia.org/T365164#9875535 (10fnegri) The alerts are visible again. I will restart the services. [14:20:26] (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110) [14:31:56] (03open) 10aborrero: kyverno: local: scale down deployment and stop requesting CPU/MEM [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/322 [14:35:11] (03update) 10aborrero: kyverno: local: scale down deployment and stop requesting CPU/MEM [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/322 [14:36:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:40:56] FIRING: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:45:24] (03update) 10aborrero: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 [14:45:29] (03merge) 10aborrero: functional-tests: build-smoke-test: retry build show test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320 [14:45:56] FIRING: [2x] SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:50:56] RESOLVED: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:51:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:55:01] (03update) 10aborrero: kyverno: local: scale down deployment and stop requesting CPU/MEM [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/322 [14:59:11] (03approved) 10dcaro: kyverno: local: scale down deployment and stop requesting CPU/MEM [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/322 (owner: 10aborrero) [15:01:18] 10Cloud Services Proposals: Decision request - kubernetes upgrade workgroup - https://phabricator.wikimedia.org/T363683#9875737 (10dcaro) This was decided, I'll wrap up soon [15:01:30] (03merge) 10aborrero: kyverno: local: scale down deployment and stop requesting CPU/MEM [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/322 [15:03:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:03:57] 06cloud-services-team: PuppetFailure Puppet failure on cloudcontrol2004-dev:9100 - https://phabricator.wikimedia.org/T367069 (10phaultfinder) 03NEW [15:08:28] 06cloud-services-team: PuppetFailure Puppet failure on cloudcontrol2004-dev:9100 - https://phabricator.wikimedia.org/T367069#9875764 (10taavi) 05Open→03Resolved a:03taavi [15:09:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:13:48] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:19:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:26:48] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudweb1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:31:48] RESOLVED: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudweb1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [16:24:34] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9876165 (10dcaro) >>! In T363983#9875492, @Andrew wrote: > If you would like to experiment with tool-specific keystone auth, I've set up an example account. The creds for... [16:39:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:49:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:17:03] 10Tool-schedule-deployment: Leave a comment on the Gerrit change when it is scheduled for a backport - https://phabricator.wikimedia.org/T366763#9876468 (10bd808) >>! In T366763#9874551, @kostajh wrote: >>>! In T366763#9868428, @bd808 wrote: >> Is there a way to write a URL + text in a Gerrit comment so that it... [17:31:28] (03update) 10dcaro: Draft: investigate authentication [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/22 (https://phabricator.wikimedia.org/T363983) [17:52:41] 06cloud-services-team, 10Toolforge (Quota-requests): Request increased quota for editgroups Toolforge tool - https://phabricator.wikimedia.org/T367002#9876778 (10bd808) +1 This request might be a good excuse to bump the `deployments.apps` quota to `16` for all tools similar to [[https://gitlab.wikimedia.org/r... [17:59:45] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9876797 (10Andrew) > > Just to have it clear myself, correct me if I'm wrong. > > * 'tool service user' is a user in LDAP with `... [18:36:16] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/11 (owner: 10l10n-bot) [18:36:19] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/11 (owner: 10l10n-bot) [18:37:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:47:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:38:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:41:45] 10Cloud-VPS (Project-requests): Request creation of capacity-exchange VPS project - https://phabricator.wikimedia.org/T296556#9877214 (10Andrew) Howdy @awight -- this project remains unmarked on this year's abandoned project roundup. Can you confirm that it's safe to delete? [19:43:24] 10Cloud-VPS (Project-requests): Request creation of citefix VPS project - https://phabricator.wikimedia.org/T335745#9877223 (10Andrew) Hello, @Jayprakash12345. Are you still using this project? If not, please respond here so I know I can safely delete it; if the project is still of use to you that's fine but... [19:51:11] 14cloud-services-team (Kanban), 10Cloud-VPS (Project-requests): Request creation of commons-corruption-checker VPS project (2) - https://phabricator.wikimedia.org/T279246#9877252 (10Andrew) Hello! This project appears to be abandoned according to https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2024_Pur... [20:01:31] 06cloud-services-team, 10Cloud-VPS (Project-requests), 06WMDE-TechWish: Request creation of "dump-references-processor" VPS project - https://phabricator.wikimedia.org/T332777#9877279 (10Andrew) Hi, folks! Is this project still in use? Either way, please update https://wikitech.wikimedia.org/wiki/News/C... [20:08:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:11:06] 10Cloud-VPS (Project-requests): Request creation of "entity-detection" VPS project - https://phabricator.wikimedia.org/T246362#9877317 (10Andrew) Hi, folks! If this project is still in use by anyone, please mark it as such at https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2024_Purge#entity-detection [20:12:14] 10Cloud-VPS (Quota-requests): Request increased quota for "entity-detection" Cloud VPS project - https://phabricator.wikimedia.org/T271790#9877321 (10Andrew) can this project be deleted now? [20:48:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:53:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:38:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks