[00:11:26] FIRING: TfInfraTestApplyFailed: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:15:28] FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:20:28] RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:30:42] dependabot[bot] opened https://github.com/toolforge/paws/pull/433 [02:37:57] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:48:27] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9901968 (10Liz) The status says this case is open and is a high priority but it's not getting any attention from those who might be in a position to resolve this problem. I've been told that "complaining" doesn't help bu... [06:37:57] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:22:12] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07): Bring an-redacteddb1001 into service to replace clouddb1021 - https://phabricator.wikimedia.org/T365453#9902186 (10Marostegui) Thanks @BTullis - if this host is going to be replacing clouddb1021 we need to update the document... [07:22:26] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07): Bring an-redacteddb1001 into service to replace clouddb1021 - https://phabricator.wikimedia.org/T365453#9902189 (10Marostegui) We also need to include this host in zarcillo (I will do that) [08:01:43] !log taavi@cloudcumin1001 automation-framework START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:02:45] !log taavi@cloudcumin1001 automation-framework END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:02:58] !log taavi@cloudcumin1001 bub2 START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:04:01] !log taavi@cloudcumin1001 bub2 END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:04:36] !log taavi@cloudcumin1001 capacity-exchange START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:05:48] !log taavi@cloudcumin1001 capacity-exchange END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:05:58] !log taavi@cloudcumin1001 catalyst START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:08:52] !log taavi@cloudcumin1001 catalyst END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:09:12] !log taavi@cloudcumin1001 catalyst-qte START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:11:09] !log taavi@cloudcumin1001 catalyst-qte END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:11:46] !log taavi@cloudcumin1001 centralnotice-staging START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:12:53] !log taavi@cloudcumin1001 centralnotice-staging END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [08:13:13] !log taavi@cloudcumin1001 checkuser-beta-wiki START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:14:16] !log taavi@cloudcumin1001 checkuser-beta-wiki END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:14:38] !log taavi@cloudcumin1001 citefix START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:15:40] !log taavi@cloudcumin1001 citefix END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:16:18] !log taavi@cloudcumin1001 civicrm-prototype START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:19:48] !log taavi@cloudcumin1001 civicrm-prototype END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:20:09] !log taavi@cloudcumin1001 collection-alt-renderer START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:20:12] !log taavi@cloudcumin1001 collection-alt-renderer END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:20:19] !log taavi@cloudcumin1001 commons-corruption-checker START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:21:31] !log taavi@cloudcumin1001 commons-corruption-checker END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:22:16] !log taavi@cloudcumin1001 commtech START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:25:29] !log taavi@cloudcumin1001 commtech END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:26:25] !log taavi@cloudcumin1001 copypatrol START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:30:18] !log taavi@cloudcumin1001 copypatrol END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:34:03] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9902324 (10fnegri) 05Open→03In progress [08:35:26] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9902321 (10fnegri) @Liz it is getting attention by multiple people, but it's not clear what the problem is. :) It might also be related to some performance issues that started last Friday on one database (T367778). This is... [08:39:29] (03update) 10aborrero: kyverno: reintroduce resource limits [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/337 (https://phabricator.wikimedia.org/T367386) [08:41:11] (03open) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 [08:47:48] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 [08:48:12] !log taavi@cloudcumin1001 cvn START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:48:28] (03update) 10sstefanova: lima-kilo: refresh source of lima_kilo_docker_addr [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/146 (owner: 10aborrero) [08:48:53] (03approved) 10sstefanova: lima-kilo: refresh source of lima_kilo_docker_addr [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/146 (owner: 10aborrero) [08:52:56] !log taavi@cloudcumin1001 cvn END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [08:53:18] !log taavi@cloudcumin1001 cyberbot START - Cookbook wmcs.openstack.migrate_project_to_ovs [08:56:56] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 [08:59:25] !log taavi@cloudcumin1001 cyberbot END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [09:00:11] !log taavi@cloudcumin1001 dashiki START - Cookbook wmcs.openstack.migrate_project_to_ovs [09:02:30] !log taavi@cloudcumin1001 dashiki END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [09:02:51] !log taavi@cloudcumin1001 devel-stats START - Cookbook wmcs.openstack.migrate_project_to_ovs [09:04:03] !log taavi@cloudcumin1001 devel-stats END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [09:04:34] !log taavi@cloudcumin1001 deployment-prep START - Cookbook wmcs.openstack.migrate_project_to_ovs [09:10:55] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 [09:29:53] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 [09:30:56] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 [09:46:28] (03open) 10sstefanova: dev: fix bump script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/43 [09:51:12] (03open) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T363346) [09:54:25] (03update) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T363346) [10:17:07] !log taavi@cloudcumin1001 deployment-prep END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [10:18:29] !log taavi@cloudcumin1001 devtools START - Cookbook wmcs.openstack.migrate_project_to_ovs [10:31:48] !log taavi@cloudcumin1001 devtools END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [10:34:16] !log taavi@cloudcumin1001 discordbots START - Cookbook wmcs.openstack.migrate_project_to_ovs [10:35:18] !log taavi@cloudcumin1001 discordbots END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [10:35:27] !log taavi@cloudcumin1001 discourse START - Cookbook wmcs.openstack.migrate_project_to_ovs [10:37:46] !log taavi@cloudcumin1001 discourse END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [10:37:51] !log taavi@cloudcumin1001 duct START - Cookbook wmcs.openstack.migrate_project_to_ovs [10:37:57] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:44:18] !log taavi@cloudcumin1001 duct END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [10:46:10] !log taavi@cloudcumin1001 dumps START - Cookbook wmcs.openstack.migrate_project_to_ovs [10:48:48] !log taavi@cloudcumin1001 dumps END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [10:50:14] 10Data-Services: [toolsdb] Clean up users and manage as code - https://phabricator.wikimedia.org/T367772#9902787 (10taavi) At least all of the 10/8 grants are outdated, anything in that range has no direct connectivity to ToolsDB. [10:50:46] !log taavi@cloudcumin1001 etytree START - Cookbook wmcs.openstack.migrate_project_to_ovs [10:50:56] !log taavi@cloudcumin1001 etytree END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [10:51:03] !log taavi@cloudcumin1001 eventmetrics START - Cookbook wmcs.openstack.migrate_project_to_ovs [10:52:23] !log taavi@cloudcumin1001 eventmetrics END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [10:54:11] !log taavi@cloudcumin1001 extdist START - Cookbook wmcs.openstack.migrate_project_to_ovs [10:55:25] !log taavi@cloudcumin1001 extdist END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [10:57:15] !log taavi@cloudcumin1001 fa-wp START - Cookbook wmcs.openstack.migrate_project_to_ovs [10:59:36] !log taavi@cloudcumin1001 fa-wp END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [10:59:58] !log taavi@cloudcumin1001 fastcci START - Cookbook wmcs.openstack.migrate_project_to_ovs [11:17:10] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services: Cloud VPS "packaging" project Buster deprecation - https://phabricator.wikimedia.org/T367544#9902873 (10MoritzMuehlenhoff) [11:31:17] (03merge) 10aborrero: lima-kilo: refresh source of lima_kilo_docker_addr [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/146 [11:33:09] (03update) 10aborrero: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 (owner: 10sstefanova) [11:33:32] (03approved) 10aborrero: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 (owner: 10sstefanova) [11:36:15] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch, 13Patch-For-Review: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9902922 (10Ladsgroup) ` ladsgroup@codesearch9:~$ sudo service hound-search status ●... [11:38:13] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch, 13Patch-For-Review: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9902936 (10Ladsgroup) Yup: ` ladsgroup@codesearch8:~$ df -h Filesystem Size U... [11:54:02] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 [12:03:53] (03PS1) 10Slyngshede: R:idp_test MPIC went away. [labs/private] - 10https://gerrit.wikimedia.org/r/1047062 [12:04:46] (03CR) 10Slyngshede: "Triggers PCC error, due to the remaining service configuration being missing." [labs/private] - 10https://gerrit.wikimedia.org/r/1047062 (owner: 10Slyngshede) [12:05:35] (03CR) 10Muehlenhoff: [C:03+1] R:idp_test MPIC went away. [labs/private] - 10https://gerrit.wikimedia.org/r/1047062 (owner: 10Slyngshede) [12:06:05] (03CR) 10Slyngshede: [C:03+2] R:idp_test MPIC went away. [labs/private] - 10https://gerrit.wikimedia.org/r/1047062 (owner: 10Slyngshede) [12:06:08] (03CR) 10Slyngshede: [V:03+2 C:03+2] R:idp_test MPIC went away. [labs/private] - 10https://gerrit.wikimedia.org/r/1047062 (owner: 10Slyngshede) [12:06:52] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 [12:16:11] (03open) 10aborrero: basic_system: add lima-kilo-boot.service [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/148 [12:22:34] (03update) 10sstefanova: dev: add pre-commit [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/147 [12:23:18] (03update) 10aborrero: basic_system: add lima-kilo-boot.service [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/148 [12:25:00] (03update) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T363346) [12:34:22] (03update) 10sstefanova: d/changelog: bump to 16.0.12 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/42 (https://phabricator.wikimedia.org/T366674) [12:34:42] (03merge) 10sstefanova: d/changelog: bump to 16.0.12 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/42 (https://phabricator.wikimedia.org/T366674) [12:41:32] 10Toolforge (Toolforge iteration 11): [jobs-api,jobs-cli] Support services in jobs - https://phabricator.wikimedia.org/T348758#9903122 (10Raymond_Ndibe) yes docs. Will add that [12:48:32] 10Cloud-VPS (Debian Buster Deprecation), 06Machine-Learning-Team: Cloud VPS "machine-learning" project Buster deprecation - https://phabricator.wikimedia.org/T367537#9903162 (10elukey) [13:20:33] (03update) 10sstefanova: d/changelog: bump to 16.0.12 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/42 (https://phabricator.wikimedia.org/T366674) [13:22:58] (03update) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T363346) [13:23:12] (03update) 10sstefanova: dev: fix bump script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/43 [13:23:19] (03update) 10sstefanova: dev: fix bump script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/43 [13:46:50] 10Cloud-VPS (Quota-requests), 10VPS-project-Codesearch: Extra 80GB volume to allow migration of buster VM to bullseye - https://phabricator.wikimedia.org/T367878 (10Ladsgroup) 03NEW [13:47:12] 10Cloud-VPS (Quota-requests), 10VPS-project-Codesearch: Extra 80GB volume to allow migration of buster VM to bullseye - https://phabricator.wikimedia.org/T367878#9903387 (10Ladsgroup) [13:47:14] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch, 13Patch-For-Review: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9903386 (10Ladsgroup) [13:49:35] !log taavi@cloudcumin1001 fastcci END (ERROR) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=97) [13:53:57] (03PS1) 10Majavah: openstack: Reduce tries in wait_for_status [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1047091 [13:54:09] !log taavi@cloudcumin1001 fastcci START - Cookbook wmcs.openstack.migrate_project_to_ovs [13:58:17] !log taavi@cloudcumin1001 fastcci END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [13:58:34] !log taavi@cloudcumin1001 foundationmemory START - Cookbook wmcs.openstack.migrate_project_to_ovs [13:59:09] (03CR) 10Majavah: [C:03+2] openstack: Reduce tries in wait_for_status [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1047091 (owner: 10Majavah) [13:59:27] !log taavi@cloudcumin1001 foundationmemory END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [14:00:03] !log taavi@cloudcumin1001 fr-tech-dev START - Cookbook wmcs.openstack.migrate_project_to_ovs [14:01:16] !log taavi@cloudcumin1001 fr-tech-dev END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [14:02:10] (03Merged) 10jenkins-bot: openstack: Reduce tries in wait_for_status [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1047091 (owner: 10Majavah) [14:15:21] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "rcm" project Buster deprecation - https://phabricator.wikimedia.org/T367548#9903488 (10Aklapper) [14:15:54] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [jobs-api, jobs-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9903494 (10Slst2020) [14:16:14] 10Toolforge (Toolforge iteration 11): [envvars-api, envvars-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363809#9903497 (10Slst2020) [14:16:49] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [builds-api, builds-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363808#9903504 (10Slst2020) [14:17:06] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikidumpparse" project Buster deprecation - https://phabricator.wikimedia.org/T367561#9903499 (10Aklapper) [14:18:06] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikicommunityhealth" project Buster deprecation - https://phabricator.wikimedia.org/T367560#9903507 (10Aklapper) [14:19:31] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikipathways" project Buster deprecation - https://phabricator.wikimedia.org/T367563#9903533 (10Aklapper) [14:22:15] 10Tool-bridgebot: Bridge IRC's cvn-sw-spam to Telegram - https://phabricator.wikimedia.org/T367884 (10Gerges) 03NEW [14:30:03] (03update) 10aborrero: basic_system: add lima-kilo-boot.service [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/148 [14:30:08] (03update) 10aborrero: basic_system: add lima-kilo-boot.service [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/148 [14:34:54] 10Tool-bridgebot: Bridge IRC's cvn-sw-spam to Telegram - https://phabricator.wikimedia.org/T367884#9903604 (10Gerges) a:03Gerges [14:37:57] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:38:44] (03open) 10sstefanova: consolidate prefixes [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 [14:43:39] (03update) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T363346) [14:44:57] (03open) 10gergesshamon: bridge IRC #cvn-sw-spam to Telegram [toolforge-repos/bridgebot] - 10https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/6 [14:47:10] (03update) 10sstefanova: consolidate prefixes [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 [14:50:41] 10Cloud-VPS (Debian Buster Deprecation), 06Machine-Learning-Team: Cloud VPS "machine-learning" project Buster deprecation - https://phabricator.wikimedia.org/T367537#9903667 (10calbon) a:03klausman [14:51:13] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9903671 (10Wurgl) @fnegri: My query had this problem since June 6th. [14:51:45] (03update) 10sstefanova: consolidate prefixes [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 [14:52:02] 10Cloud-VPS (Quota-requests), 10VPS-project-Codesearch: Extra 80GB volume to allow migration of buster VM to bullseye - https://phabricator.wikimedia.org/T367878#9903674 (10Andrew) +1 Thanks for working on OS migration! [14:54:24] !log taavi@cloudcumin1001 codesearch START - Cookbook wmcs.openstack.quota_increase (T367878) [14:54:29] T367878: Extra 80GB volume to allow migration of buster VM to bullseye - https://phabricator.wikimedia.org/T367878 [14:54:32] !log taavi@cloudcumin1001 codesearch END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T367878) [14:55:01] 10Cloud-VPS (Quota-requests), 10VPS-project-Codesearch: Extra 80GB volume to allow migration of buster VM to bullseye - https://phabricator.wikimedia.org/T367878#9903683 (10taavi) 05Open→03Resolved a:03taavi [15:02:22] (03update) 10gergesshamon: bridge IRC #cvn-sw-spam to Telegram [toolforge-repos/bridgebot] - 10https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/6 [15:02:44] (03update) 10aborrero: kind: add additional worker nodes to kubernetes [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/139 [15:04:29] (03update) 10aborrero: kind: add additional worker nodes to kubernetes [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/139 [15:04:46] (03merge) 10aborrero: kind: add additional worker nodes to kubernetes [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/139 [15:06:47] (03close) 10gergesshamon: bridge IRC #cvn-sw-spam to Telegram [toolforge-repos/bridgebot] - 10https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/6 [15:07:32] !log taavi@cloudcumin1001 glams START - Cookbook wmcs.openstack.migrate_project_to_ovs [15:08:50] !log taavi@cloudcumin1001 glams END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [15:09:03] !log taavi@cloudcumin1001 glamwikidashboard START - Cookbook wmcs.openstack.migrate_project_to_ovs [15:10:18] 06cloud-services-team, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#9903770 (10Pppery) [15:10:31] 06cloud-services-team, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#9903771 (10Pppery) [15:11:25] !log taavi@cloudcumin1001 glamwikidashboard END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [15:13:08] (03open) 10gergesshamon: bridge IRC #cvn-sw-spam to Telegram [toolforge-repos/bridgebot] - 10https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/7 [15:13:47] (03update) 10sstefanova: consolidate prefixes [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 [15:15:47] (03update) 10gergesshamon: bridge IRC #cvn-sw-spam to Telegram [toolforge-repos/bridgebot] - 10https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/7 [15:20:36] (03open) 10taavi: cloudvps_flavors: Add g4.cores8.ram24.disk20.ephemeral40.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/5 [15:21:09] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [wikireplicas] frequent replag spikes in clouddb1017 (s1) - https://phabricator.wikimedia.org/T367778#9903801 (10fnegri) [15:21:44] (03PS1) 10Majavah: openstack: Add g3.cores8.ram24.disk20.ephemeral40.4xiops to allowlist [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1047109 [15:22:02] (03approved) 10andrew: cloudvps_flavors: Add g4.cores8.ram24.disk20.ephemeral40.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/5 (owner: 10taavi) [15:22:21] (03merge) 10taavi: cloudvps_flavors: Add g4.cores8.ram24.disk20.ephemeral40.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/5 [15:23:04] (03CR) 10Majavah: [C:03+2] openstack: Add g3.cores8.ram24.disk20.ephemeral40.4xiops to allowlist [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1047109 (owner: 10Majavah) [15:24:20] (03update) 10sstefanova: Draft: consolidate prefixes [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 [15:25:49] !log taavi@cloudcumin1001 gitlab-runners START - Cookbook wmcs.openstack.migrate_server_to_ovs for server gitlab-runners-puppetserver-01 [15:26:10] (03Merged) 10jenkins-bot: openstack: Add g3.cores8.ram24.disk20.ephemeral40.4xiops to allowlist [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1047109 (owner: 10Majavah) [15:26:58] !log taavi@cloudcumin1001 gitlab-runners END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server gitlab-runners-puppetserver-01 [15:27:16] !log taavi@cloudcumin1001 gitlab-runners START - Cookbook wmcs.openstack.migrate_project_to_ovs [15:36:22] andrewbogott opened https://github.com/toolforge/quarry/pull/57 [15:37:51] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate Cloud VPS to Neutron Open vSwitch agent - https://phabricator.wikimedia.org/T326373#9903856 (10github-toolforge-bot) andrewbogott opened https://github.com/toolforge/quarry/pull/57 [15:41:22] !log taavi@cloudcumin1001 gitlab-runners END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [15:46:25] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch, 13Patch-For-Review: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9903894 (10Ladsgroup) attached the new volume, it's being rebuilt. [15:48:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:53:38] RESOLVED: [4x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:54:08] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: server fails to reboot for clouddb1018.eqiad.wmnet - https://phabricator.wikimedia.org/T367499#9903941 (10fnegri) Thanks @Jclark-ctr! The host is now repooled. [15:59:48] !log taavi@cloudcumin1001 globaleducation START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:00:05] 10Cloud-VPS: dwl reboot coordination request - https://phabricator.wikimedia.org/T367797#9903974 (10taavi) a:05taavi→03Andrew [16:07:32] !log taavi@cloudcumin1001 globaleducation END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [16:07:54] !log taavi@cloudcumin1001 gratitude START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:07:56] !log taavi@cloudcumin1001 gratitude END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:08:05] !log taavi@cloudcumin1001 hashtags START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:09:26] !log taavi@cloudcumin1001 hashtags END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:09:43] !log taavi@cloudcumin1001 hoiscript START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:10:54] !log taavi@cloudcumin1001 hoiscript END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:12:21] !log taavi@cloudcumin1001 huggle START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:14:42] !log taavi@cloudcumin1001 huggle END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:16:06] !log taavi@cloudcumin1001 huma START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:17:18] !log taavi@cloudcumin1001 huma END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:18:09] !log taavi@cloudcumin1001 iiab START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:19:33] 06cloud-services-team, 10Toolforge, 13Patch-For-Review, 10Sustainability (Incident Followup): [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways - https://phabricator.wikimedia.org/T367386#9904103 (10aborrero) I tested this: * increased the number of cluster nodes in lima... [16:21:30] !log taavi@cloudcumin1001 iiab END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:23:59] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [wikireplicas] frequent replag spikes in clouddb1017 (s1) - https://phabricator.wikimedia.org/T367778#9904118 (10fnegri) The lag grows until about 3 hours, then starts decreasing. This is consistent with wmf-pt-kill that is configured to kill queries... [16:25:40] !log taavi@cloudcumin1001 image-suggestion-api START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:25:43] !log taavi@cloudcumin1001 image-suggestion-api END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:26:30] !log taavi@cloudcumin1001 imagebulk START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:27:48] !log taavi@cloudcumin1001 imagebulk END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:30:41] !log taavi@cloudcumin1001 imagehash START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:31:43] !log taavi@cloudcumin1001 imagehash END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:37:06] !log taavi@cloudcumin1001 impactvisualizer START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:37:09] !log taavi@cloudcumin1001 impactvisualizer END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:40:35] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [wikireplicas] frequent replag spikes in clouddb1017 (s1) - https://phabricator.wikimedia.org/T367778#9904188 (10fnegri) As suggested by @taavi I tried depooling `s1` on `clouddb1017`, so that all `s1` wikireplica traffic will go to the other host (`c... [16:47:22] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [wikireplicas] frequent replag spikes in clouddb1017 (s1) - https://phabricator.wikimedia.org/T367778#9904219 (10fnegri) > Out of the total 170 queries killed, 69 include /* pollcats.rs SLOW_OK */ These queries are made by the user `s51073` which map... [16:48:16] 06cloud-services-team, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#9904220 (10Pppery) [16:48:28] 06cloud-services-team, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#9904221 (10Pppery) [16:48:38] 06cloud-services-team, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#9904222 (10Pppery) Is this finally done now? [16:49:41] 10Quarry: github action building main rather than branch - https://phabricator.wikimedia.org/T367630#9904225 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/58 [16:49:50] vivian-rook opened https://github.com/toolforge/quarry/pull/58 [16:51:45] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:52:51] 10Quarry: github action building main rather than branch - https://phabricator.wikimedia.org/T367630#9904248 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/58 [16:53:01] vivian-rook closed https://github.com/toolforge/quarry/pull/58 [16:53:59] vivian-rook opened https://github.com/toolforge/quarry/pull/59 [16:56:45] RESOLVED: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:58:49] vivian-rook closed https://github.com/toolforge/quarry/pull/59 [17:00:03] 10Quarry: github action building main rather than branch - https://phabricator.wikimedia.org/T367630#9904269 (10rook) 05Open→03Resolved a:03rook [17:04:56] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9904291 (10fnegri) > If you look through Execution time column on Recent queries list, it actually seems like that results of virtually any query with execution time longer than ~120s will never make it back I had to scrol... [17:16:39] andrewbogott opened https://github.com/toolforge/quarry/pull/60 [17:22:19] !log andrew@cloudcumin1001 dwl START - Cookbook wmcs.openstack.migrate_project_to_ovs [17:23:51] !log andrew@cloudcumin1001 dwl END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [17:30:44] (03PS1) 10Pppery: Archive repo [labs/tools/dawiki] - 10https://gerrit.wikimedia.org/r/1047146 (https://phabricator.wikimedia.org/T270105) [17:48:05] !log andrew@cloudcumin1001 dwl START - Cookbook wmcs.openstack.migrate_project_to_ovs [17:52:50] !log andrew@cloudcumin1001 dwl END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [18:37:57] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:10:23] FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM [19:15:23] RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM [19:36:23] andrewbogott closed https://github.com/toolforge/quarry/pull/60 [19:42:00] andrewbogott opened https://github.com/toolforge/paws/pull/434 [20:48:46] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate Cloud VPS to Neutron Open vSwitch agent - https://phabricator.wikimedia.org/T326373#9905141 (10github-toolforge-bot) andrewbogott closed https://github.com/toolforge/paws/pull/434 [20:48:46] andrewbogott closed https://github.com/toolforge/paws/pull/434 [20:52:35] andrewbogott opened https://github.com/toolforge/superset-deploy/pull/22 [20:52:35] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate Cloud VPS to Neutron Open vSwitch agent - https://phabricator.wikimedia.org/T326373#9905148 (10github-toolforge-bot) andrewbogott opened https://github.com/toolforge/superset-deploy/pull/22 [20:53:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D{cloudcontrol1006.eqiad.wmnet}' [20:55:22] FIRING: [10x] HAProxyBackendUnavailable: HAProxy service designate-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [20:55:23] PROBLEM - Host cloudcontrol1006 is DOWN: PING CRITICAL - Packet loss = 100% [20:56:10] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [20:58:08] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D{cloudcontrol1006.eqiad.wmnet}' [20:58:26] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:58:29] RECOVERY - Host cloudcontrol1006 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [20:59:37] 10superset.wmcloud.org: Upgrade to 4.0.0 - https://phabricator.wikimedia.org/T364022#9905184 (10github-toolforge-bot) andrewbogott closed https://github.com/toolforge/superset-deploy/pull/21 [20:59:49] andrewbogott closed https://github.com/toolforge/superset-deploy/pull/21 [21:00:22] FIRING: [14x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:01:10] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [21:04:41] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate Cloud VPS to Neutron Open vSwitch agent - https://phabricator.wikimedia.org/T326373#9905202 (10github-toolforge-bot) andrewbogott opened https://github.com/toolforge/superset-deploy/pull/23 [21:04:47] andrewbogott opened https://github.com/toolforge/superset-deploy/pull/23 [21:05:22] RESOLVED: [14x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:07:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [21:18:36] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [21:18:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:20:41] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudvirt-wdqs100[1,2,3] - https://phabricator.wikimedia.org/T367773#9905260 (10VRiley-WMF) a:03VRiley-WMF [21:25:30] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudvirt-wdqs100[1,2,3] - https://phabricator.wikimedia.org/T367773#9905281 (10VRiley-WMF) [21:32:43] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudvirt-wdqs100[1,2,3] - https://phabricator.wikimedia.org/T367773#9905314 (10VRiley-WMF) 05Open→03Resolved [21:50:39] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9905354 (10Dzahn) I deleted the webproxy `codesearch-old.wmcloud.org`. I think we can also delete `codes... [21:56:20] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9905397 (10Dzahn) >>! In T367479#9904556, @Ladsgroup wrote: > FWIW, I didn't create a systemd service for... [21:58:54] 10VPS-project-Codesearch, 06collaboration-services: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#9905399 (10Dzahn) One thing to be done here: add a systemd unit for the frontend. See T367479#9904556 [22:05:03] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9905404 (10Dzahn) Since the buster machine is already shut down this is resolved. Thanks Ladsgroup! [22:54:05] 10Tool-bridgebot: Bridge IRC's cvn-sw-spam to Telegram - https://phabricator.wikimedia.org/T367884#9905511 (10LucasWerkmeister) I looked a bit into this, but I’m not sure it’s a good idea, to be honest. This would be a very high-volume bridge, at least compared to the other bridged channels I’m aware of, and I’m... [23:53:30] (03update) 10raymond-ndibe: [jobs-api] move simple job validations to pydantic [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/89 (https://phabricator.wikimedia.org/T366209)