[00:01:49] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335 (10Mhurd) 03NEW [00:04:00] (03approved) 10raymond-ndibe: [volume-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/21 (https://phabricator.wikimedia.org/T362867) [00:04:05] (03merge) 10raymond-ndibe: [volume-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/21 (https://phabricator.wikimedia.org/T362867) [00:04:22] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10338784 (10Mhurd) [00:06:48] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: volume-admission: bump to 0.0.59-20241120000415-71c39564 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/610 (https://phabricator.wikimedia.org/T362867) [00:09:39] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T362867) [00:09:43] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [00:11:49] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10338808 (10Mhurd) [00:12:27] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10338809 (10Mhurd) [00:13:34] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10338811 (10Mhurd) [00:14:36] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10338812 (10Mhurd) [00:14:52] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [00:15:31] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T362867) [00:15:36] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [00:16:18] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T362867) [00:22:49] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T362867) [00:22:54] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [00:23:42] (03approved) 10raymond-ndibe: volume-admission: bump to 0.0.59-20241120000415-71c39564 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/610 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:23:47] (03merge) 10raymond-ndibe: volume-admission: bump to 0.0.59-20241120000415-71c39564 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/610 (https://phabricator.wikimedia.org/T362867) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:24:34] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10338825 (10Raymond_Ndibe) [00:33:36] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10338828 (10Andrew) I'm not sure how rabbitmq got into this state but I explicitly had it forget a pool member and reset everything and it seems to be working properly now. T... [00:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:28:57] 10tool-wscontest, 07good first task: Add UTC in the WSContest contest page - https://phabricator.wikimedia.org/T331225#10338847 (10Samwilson) [02:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:38:18] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10338909 (10Mhurd) [02:57:03] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: Upgrade cloud-vps openstack to version 'Caracal' - https://phabricator.wikimedia.org/T369044#10338919 (10Andrew) 05Open→03Resolved [02:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:59:55] 06cloud-services-team, 10Cloud-VPS: codfw1dev: rabbitmq is not working because some auth failures - https://phabricator.wikimedia.org/T374002#10338923 (10Andrew) 05Open→03Resolved I believe this is resolved, reopen if necessary. [03:14:49] 06cloud-services-team, 10Cloud-VPS, 10Ceph: puppet: partman comments in cephosd.cfg are misleading - https://phabricator.wikimedia.org/T380339 (10Andrew) 03NEW [03:18:02] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825#10338946 (10Andrew) a:05Andrew→03None puppet is updated (although untested, for obvious reasons) [03:32:26] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Openstack: many orphaned (or seemingly orphaned) VMs in codfw1dev - https://phabricator.wikimedia.org/T380303#10338955 (10Andrew) 05Open→03Resolved [03:36:52] 10tool-wscontest, 07good first task: Add UTC in the WSContest contest page - https://phabricator.wikimedia.org/T331225#10338958 (10Samwilson) Thanks @AS1100K, manually mentioning PRs is the correct way to do it now. There's no automatic tracking of GitHub PRs here; that only works for Gerrit and Wikimedia's Gi... [05:14:06] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [05:14:48] (03open) 10raymond-ndibe: [builds-builder] create toolforge harbor project [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/63 (https://phabricator.wikimedia.org/T358225) [05:21:43] (03open) 10raymond-ndibe: [maintain-harbor] comment about the importance of some logs in tests [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/38 (https://phabricator.wikimedia.org/T358225) [05:22:57] (03update) 10raymond-ndibe: [builds-builder] create toolforge harbor project [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/63 (https://phabricator.wikimedia.org/T358225) [05:23:22] (03update) 10raymond-ndibe: [builds-builder] create toolforge harbor project [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/63 (https://phabricator.wikimedia.org/T358225) [05:28:19] (03update) 10raymond-ndibe: [lima-kilo] support maintain-harbor functional tests [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/214 (https://phabricator.wikimedia.org/T358225) [05:29:21] (03update) 10raymond-ndibe: [lima-kilo] add harbor to toolforge-common.yaml [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/214 (https://phabricator.wikimedia.org/T358225) [08:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:21:32] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10339076 (10dcaro) +1 approved [08:34:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:35:42] (03approved) 10dcaro: [maintain-harbor] comment about the importance of some logs in tests [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/38 (https://phabricator.wikimedia.org/T358225) (owner: 10raymond-ndibe) [08:38:12] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade to v16 - https://phabricator.wikimedia.org/T306820#10339107 (10dcaro) a:03dcaro This one is still needed yes, and we should push it next quarter [08:43:03] 10cloud-services-team (FY2024/2025-Q1-Q2): Drain C8 rack - https://phabricator.wikimedia.org/T374043#10339112 (10dcaro) 05Open→03Resolved No, this is done [08:50:15] (03approved) 10sstefanova: utils: copy scripts from webservice [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/34 (owner: 10dcaro) [09:00:04] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10339157 (10Peter) [09:58:33] 10ToolforgeBundle: Upgrade to Symfony 7 - https://phabricator.wikimedia.org/T361554#10339259 (10Samwilson) Another issue when upgrading: > $ ./bin/console c:c > [WARNING] Some commands could not be registered: > The command defined in "Wikim... [09:59:43] (03open) 10aborrero: eqiad1: create project mediawiki-quickstart [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/133 (https://phabricator.wikimedia.org/T380335) [10:01:29] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [10:01:50] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [10:04:11] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:05:26] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:07:12] (03update) 10dcaro: utils: copy scripts from webservice [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/34 [10:07:24] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10339274 (10fnegri) > having it delete things that are undefined in opentofu is generally bad. In general, Tofu never deletes resources it doesn't know about. In T380310, the projects... [10:08:01] (03update) 10dcaro: utils: copy scripts from webservice [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/34 [10:08:17] (03approved) 10dcaro: utils: copy scripts from webservice [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/34 [10:08:34] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10339275 (10fnegri) p:05Triage→03Medium [10:08:40] (03merge) 10dcaro: utils: copy scripts from webservice [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/34 [10:11:49] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10339284 (10aborrero) >>! In T380310#10339274, @fnegri wrote: >> we can easily prevent tofu from deleting resources: >> https://opentofu.org/docs/v1.7/language/meta-arguments/lifecycle/... [10:12:53] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10339286 (10aborrero) [10:13:16] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [10:13:41] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [10:16:22] (03update) 10aborrero: eqiad1: create project mediawiki-quickstart [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/133 (https://phabricator.wikimedia.org/T380335) [10:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:26:22] (03approved) 10aborrero: eqiad1: create project mediawiki-quickstart [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/133 (https://phabricator.wikimedia.org/T380335) [10:26:23] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): unable to log-in to toolsbeta-harbor-1 - https://phabricator.wikimedia.org/T379633#10339327 (10fnegri) Puppet was failing on this host because of {T379927}. I manually added the missing `nameserver` line to `resolv.conf` and Puppet is now working... [10:26:28] (03merge) 10aborrero: eqiad1: create project mediawiki-quickstart [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/133 (https://phabricator.wikimedia.org/T380335) [10:26:37] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:26:40] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): unable to log-in to toolsbeta-harbor-1 - https://phabricator.wikimedia.org/T379633#10339321 (10dcaro) 05Open→03Resolved a:03dcaro [10:26:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): unable to log-in to toolsbeta-harbor-1 - https://phabricator.wikimedia.org/T379633#10339325 (10dcaro) This was another instance of https://phabricator.wikimedia.org/T379927, thanks @fnegri for fixing it [10:27:35] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:31:24] (03update) 10dcaro: global: add extra envvars configuration [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/15 (https://phabricator.wikimedia.org/T347141) [10:33:52] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10339358 (10zeljkofilipin) [10:35:05] 10Cloud-VPS (Project-requests): Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10339343 (10aborrero) 05Open→03Resolved a:03aborrero [10:40:24] (03approved) 10sstefanova: global: add extra envvars configuration [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/15 (https://phabricator.wikimedia.org/T347141) (owner: 10dcaro) [10:44:34] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10339414 (10aborrero) Maybe we can use the wmfkeystonehook mechanism to prevent deletions for projects if they would have leaked resources. I'm not sure if we can 'abort' a deletion op... [10:46:55] (03merge) 10dcaro: global: add extra envvars configuration [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/15 (https://phabricator.wikimedia.org/T347141) [10:48:38] (03open) 10dcaro: README: updated the development lifecycle diagram [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/611 [10:49:36] (03update) 10dcaro: [toolforge-deploy] allow for running both admin and tools tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/605 (https://phabricator.wikimedia.org/T358225) (owner: 10raymond-ndibe) [10:51:24] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: envvars-admission: bump to 0.0.22-20241120104704-1c22695e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/612 (https://phabricator.wikimedia.org/T347141) [10:53:17] (03approved) 10fnegri: README: updated the development lifecycle diagram [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/611 (owner: 10dcaro) [10:56:13] (03update) 10dcaro: envvars-admission: bump to 0.0.22-20241120104704-1c22695e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/612 (https://phabricator.wikimedia.org/T347141) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:56:34] (03merge) 10dcaro: README: updated the development lifecycle diagram [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/611 [11:09:31] 06cloud-services-team, 10Cloud-VPS, 10VPS-Projects: Quota leak for Trove dbs? - https://phabricator.wikimedia.org/T373348#10339480 (10fnegri) The "reserved" column is inconsistent again, but this time it's `ram` and `volumes` that were not reset, instead of `instances` and `volumes`: ` fnegri@cloudcontrol10... [11:13:53] 06cloud-services-team, 10Cloud-VPS, 10VPS-Projects: Quota leak for Trove dbs? - https://phabricator.wikimedia.org/T373348#10339489 (10fnegri) The `reservations` table looks consistent: ` mysql:root@localhost [trove_eqiad1]> select * from reservations where created>'2024-11-19' and usage_id = "ccd3350c-338d-... [11:23:05] 10VPS-project-Codesearch, 10dev-images: Add releng/dev-images to codesearch - https://phabricator.wikimedia.org/T380133#10339519 (10Daimona) 05Open→03Resolved [11:40:53] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10339577 (10aborrero) there was a problem with the bastion ssh key: `lang=shell-session aborrero@cloudcontrol2005-dev:~$ sudo /usr/bin/ssh -o ConnectTimeout=5 -o UserKnownHos... [11:43:18] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: [wmcs-cookbooks] wmcs.openstack.cloudvirt.vm_console cookbook is not working from cloudcumin hosts - https://phabricator.wikimedia.org/T379570#10339579 (10fnegri) a:03fnegri [11:51:20] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10339598 (10aborrero) another problem, the puppetmaster is not accepting the SSH connection from designate wmfsink for cleanup: `lang=shell-session aborrero@cloudcontrol2005-... [11:52:47] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10339604 (10aborrero) server side logs: ` 2024-11-20T11:52:00.958571+00:00 cloudinfra-cloudvps-puppetserver-1 sshd[3821940]: Connection from 172.20.5.7 port 54050 on 172.16.1... [11:58:00] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [12:00:30] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10339634 (10aborrero) There could be problems with the puppet role. * In eqiad1, project `cloudinfra`, puppet prefix `cloudinfra-cloudvps-puppetserver` has role `role::puppet... [12:02:56] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [12:08:56] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10339678 (10aborrero) Corrected discrepancy in the puppet role by using `role::puppetserver::cloud_vps_global` on the codfw1dev prefix. The puppetserver still rejects the ssh... [12:09:26] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [12:11:26] (03open) 10pwangai: Optimize SonarQube Bot Reports by Linking Only Relevant [toolforge-repos/sonarqubebot-experimental] - 10https://gitlab.wikimedia.org/toolforge-repos/sonarqubebot-experimental/-/merge_requests/4 (https://phabricator.wikimedia.org/T380355) [12:13:03] (03merge) 10pwangai: Optimize SonarQube Bot Reports by Linking Only Relevant [toolforge-repos/sonarqubebot-experimental] - 10https://gitlab.wikimedia.org/toolforge-repos/sonarqubebot-experimental/-/merge_requests/4 (https://phabricator.wikimedia.org/T380355) [12:15:31] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [12:19:42] (03approved) 10dcaro: envvars-admission: bump to 0.0.22-20241120104704-1c22695e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/612 (https://phabricator.wikimedia.org/T347141) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:19:47] (03update) 10dcaro: envvars-admission: bump to 0.0.22-20241120104704-1c22695e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/612 (https://phabricator.wikimedia.org/T347141) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:20:09] (03merge) 10dcaro: envvars-admission: bump to 0.0.22-20241120104704-1c22695e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/612 (https://phabricator.wikimedia.org/T347141) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:23:27] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10339769 (10aborrero) >>! In T380208#10339678, @aborrero wrote: > Corrected discrepancy in the puppet role by using `role::puppetserver::cloud_vps_global` on the codfw1dev pre... [12:24:59] (03update) 10dcaro: [toolforge-deploy] allow for running both admin and tools tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/605 (https://phabricator.wikimedia.org/T358225) (owner: 10raymond-ndibe) [12:31:10] (03update) 10dcaro: toolforge_deploy_mr: add clis to the bash completion [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/215 [12:44:12] (03update) 10dcaro: [builds-builder] create toolforge harbor project [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/63 (https://phabricator.wikimedia.org/T358225) (owner: 10raymond-ndibe) [12:54:55] (03open) 10aborrero: codfw1dev: admin-monitoring: track default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/134 (https://phabricator.wikimedia.org/T380208) [12:56:09] (03merge) 10aborrero: codfw1dev: admin-monitoring: track default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/134 (https://phabricator.wikimedia.org/T380208) [12:56:16] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:56:45] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:57:23] 10Toolforge (Toolforge iteration 16): [components-api] Limit the amount of deployments to (say) 25 - https://phabricator.wikimedia.org/T380283#10339938 (10dcaro) a:03dcaro [12:57:53] 10Toolforge (Toolforge iteration 16): [components-api] Limit the amount of deployments to (say) 25 - https://phabricator.wikimedia.org/T380283#10339940 (10dcaro) 05Open→03In progress [12:58:11] (03open) 10dcaro: deployment: cleanup old deployment on deployment creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T380283) [12:59:51] (03update) 10dcaro: deployment: cleanup old deployment on deployment creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T380283) [13:06:38] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10339982 (10aborrero) the VMs get the right hostname, and can run puppet correctly: `lang=shell-session root@fullstackd-20241120124330:~# run-puppet-age... [13:17:39] (03update) 10dcaro: deployment: cleanup old deployment on deployment creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T380283) [13:21:05] (03open) 10dcaro: deploy_task: use patch when creating the job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/40 [13:23:00] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10340016 (10aborrero) I have observed the following: * if you cleanup everything by hand (VMs, DNS records, etc) * then the first VM will pass all the c... [13:23:46] (03update) 10dcaro: deploy_task: use patch when creating the job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/40 [13:26:25] (03update) 10dcaro: deploy_task: use patch when creating the job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/40 [13:27:26] (03update) 10dcaro: deploy_task: use patch when creating the job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/40 [13:27:42] (03update) 10dcaro: deploy_task: use patch when creating the job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/40 [13:28:05] (03open) 10dcaro: add coverage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/41 [13:28:34] (03update) 10dcaro: add coverage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/41 [13:29:00] (03update) 10dcaro: add coverage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/41 [13:40:08] (03update) 10dcaro: deployment: cleanup old deployment on deployment creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T380283) [13:42:45] (03open) 10dcaro: envvars-admission: add elasticsearch and redis URLs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/613 (https://phabricator.wikimedia.org/T347141) [13:51:47] 10VPS-project-Codesearch, 10LPL Hypothesis: Index recommendation-api in the code search tool - https://phabricator.wikimedia.org/T379761#10340161 (10SBisson) 05Open→03Resolved The patch was merged and deployed. Results from the recommendation api can now be found: https://codesearch.wmcloud.org/searc... [13:53:48] (03open) 10dcaro: api: expose the healthz endpoint too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/53 (https://phabricator.wikimedia.org/T348633) [13:56:43] (03update) 10dcaro: api: expose the healthz endpoint too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/53 (https://phabricator.wikimedia.org/T348633) [13:57:22] (03approved) 10fnegri: envvars-admission: add elasticsearch and redis URLs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/613 (https://phabricator.wikimedia.org/T347141) (owner: 10dcaro) [13:57:30] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10340222 (10aborrero) I have the suspicion that only cloudcontrol2004-dev is executing the designate sink plugin code. [13:58:43] (03merge) 10dcaro: envvars-admission: add elasticsearch and redis URLs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/613 (https://phabricator.wikimedia.org/T347141) [13:59:27] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10340235 (10aborrero) >>! In T380208#10340222, @aborrero wrote: > I have the suspicion that only cloudcontrol2004-dev is executing the designate sink plu... [14:03:13] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [14:03:17] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [14:04:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [14:06:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#10340258 (10dcaro) 05Open→03In progress [14:08:28] (03approved) 10fnegri: add coverage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/41 (owner: 10dcaro) [14:08:48] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [14:08:52] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [14:10:28] (03approved) 10fnegri: deploy_task: use patch when creating the job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/40 (owner: 10dcaro) [14:10:32] (03merge) 10dcaro: add coverage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/41 [14:12:25] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.67-20241120141044-cb24ddb0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/614 [14:12:29] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.67-20241120141044-cb24ddb0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/614 [14:14:22] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [14:15:31] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [14:21:54] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [14:24:53] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [14:25:33] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [envvars,maintain-kubeusers] create and populate envvars for common service names - https://phabricator.wikimedia.org/T347141#10340301 (10dcaro) Deployed and updated the wiki pages [14:31:00] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [14:31:01] 10Toolforge (Toolforge iteration 16): [envvars,maintain-kubeusers] create and populate envvars for common service names - https://phabricator.wikimedia.org/T347141#10340306 (10dcaro) 05In progress→03Resolved [14:40:20] (03approved) 10dcaro: components-api: bump to 0.0.67-20241120141044-cb24ddb0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/614 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:40:22] (03merge) 10dcaro: components-api: bump to 0.0.67-20241120141044-cb24ddb0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/614 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:40:30] (03update) 10dcaro: deploy_task: use patch when creating the job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/40 [14:40:40] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10340372 (10Andrew) [14:41:57] (03merge) 10dcaro: deploy_task: use patch when creating the job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/40 [14:43:10] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10340376 (10Andrew) > In general, Tofu never deletes resources it doesn't know about. In T380310, the projects were actually defined in opentofu, but the servers inside the project we... [14:43:59] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.68-20241120144205-2811ef71 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/615 [14:44:09] (03update) 10fnegri: api: expose the healthz endpoint too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/53 (https://phabricator.wikimedia.org/T348633) (owner: 10dcaro) [14:44:11] (03approved) 10fnegri: api: expose the healthz endpoint too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/53 (https://phabricator.wikimedia.org/T348633) (owner: 10dcaro) [14:45:00] (03merge) 10dcaro: api: expose the healthz endpoint too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/53 (https://phabricator.wikimedia.org/T348633) [14:46:34] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [14:47:09] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: api-gateway: bump to 0.0.56-20241120144516-f10abf2a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/616 (https://phabricator.wikimedia.org/T348633) [14:47:13] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: api-gateway: bump to 0.0.56-20241120144516-f10abf2a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/616 (https://phabricator.wikimedia.org/T348633) [14:48:55] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10340408 (10fnegri) > simply undefined those projects which resulted in them being deleted That's correct, if you remove a resource from the Tofu configuration, and Tofu knows about th... [14:53:17] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10340420 (10fnegri) > I create an out-of-band project X using e.g. the cli > on next tofu run project X is 'cleaned up' The out-of-band project will be ignored by Tofu, unless you manu... [14:53:20] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [14:53:55] (03approved) 10dcaro: components-api: bump to 0.0.68-20241120144205-2811ef71 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/615 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:53:58] (03merge) 10dcaro: components-api: bump to 0.0.68-20241120144205-2811ef71 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/615 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:54:02] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [14:54:15] (03update) 10dcaro: api-gateway: bump to 0.0.56-20241120144516-f10abf2a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/616 (https://phabricator.wikimedia.org/T348633) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:56:11] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [15:01:40] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [15:06:08] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [15:09:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [15:15:51] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [15:17:35] (03approved) 10dcaro: api-gateway: bump to 0.0.56-20241120144516-f10abf2a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/616 (https://phabricator.wikimedia.org/T348633) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:17:38] (03merge) 10dcaro: api-gateway: bump to 0.0.56-20241120144516-f10abf2a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/616 (https://phabricator.wikimedia.org/T348633) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:22:09] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#10340560 (10dcaro) 05In progress→03Resolved [15:26:04] 10Striker: [toolsadmin] Username is already in use or invalid. - https://phabricator.wikimedia.org/T380384 (10RoyZuo) 03NEW [15:27:24] 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10340613 (10Andrew) Ok! That is not as scary as I feared. [15:41:46] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-5:30003 has failed probes (http_api_svc_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:47:36] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30003 has failed probes (http_api_svc_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:56:51] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#10340927 (10dcaro) 05Resolved→03In progress [17:01:46] (03open) 10dcaro: jobs-emailer: use strings instead of numbers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/617 [17:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:07:36] FIRING: ProbeDown: Service api.svc.beta.toolforge.org:443 has failed probes (http_api_svc_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#api.svc.beta.toolforge.org:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:08:19] 10Cloud-VPS (Project-requests), 10MediaWiki-Quickstart: Request creation of "mediawiki-quickstart" VPS project - https://phabricator.wikimedia.org/T380335#10341304 (10Mhurd) [18:12:36] RESOLVED: ProbeDown: Service api.svc.beta.toolforge.org:443 has failed probes (http_api_svc_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#api.svc.beta.toolforge.org:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:24:16] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#10341396 (10dcaro) 05In progress→03Resolved [18:28:16] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-5:30003 has failed probes (http_api_svc_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:35:17] (03approved) 10dcaro: jobs-emailer: use strings instead of numbers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/617 [18:35:20] (03merge) 10dcaro: jobs-emailer: use strings instead of numbers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/617 [18:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:04:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:02:27] 10Striker: Add one-time-code autocomplete type to TOTP field on toolsadmin login page - https://phabricator.wikimedia.org/T371794#10341949 (10bd808) 05Open→03Resolved a:03XtexChooser This was merged, but then the entire 2FA integration was removed for {T376190} as a side effect of the #wikitech.wikimed... [21:15:04] 10Tools: bullseye's Spur API key has expired - https://phabricator.wikimedia.org/T380193#10342014 (10RoySmith) I love Bullseye, but I just don't see the point of continuing to maintain it when there's a WMF-supported service that basically does the same thing. If there's something Bullseye does that IPoid doesn... [21:35:18] 10Striker: [toolsadmin] Username is already in use or invalid. - https://phabricator.wikimedia.org/T380384#10342075 (10bd808) p:05Triage→03High [21:36:37] 10Striker: [toolsadmin] Striker cannot create Developer accounts with names matching existing SUL accounts - https://phabricator.wikimedia.org/T380384#10342072 (10bd808) This is an unfortunate side effect of {T161859} having started and {T364605} not yet having been implemented. The failing validation API is ht... [21:36:38] 10Striker: [toolsadmin] Striker cannot create Developer accounts with names matching existing SUL accounts - https://phabricator.wikimedia.org/T380384#10342076 (10bd808) [21:37:40] 10Striker: [toolsadmin] Striker cannot create Developer accounts with names matching existing SUL accounts - https://phabricator.wikimedia.org/T380384#10342077 (10bd808) [22:06:39] 10Striker: [toolsadmin] Striker cannot create Developer accounts with names matching existing SUL accounts - https://phabricator.wikimedia.org/T380384#10342154 (10bd808) [22:06:45] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations: Move Striker to Bitu username validation API - https://phabricator.wikimedia.org/T364605#10342155 (10bd808) [22:07:31] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations: Move Striker to Bitu username validation API - https://phabricator.wikimedia.org/T364605#10342153 (10bd808) > Tokens can be requests via https://idm-test.wikimedia.org once the attached patch has been merged. The required patch for is... [22:30:40] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T380426 (10WaterQuark) 03NEW [22:55:02] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10342301 (10bd808) >>! In T376267#10336758, @Rodrigo wrote: > |**Wikitech account/LDAP:**| Rodrigo| > |**SUL account**| Rodrigo| > |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]*... [23:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks