[00:10:49] FIRING: TfInfraTestApplyFailed: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [01:12:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:12:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:33:42] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9891513 (10SD0001) One such case: https://quarry.wmcloud.org/query/83084 The query actually completed on the db in 140 seconds but there was a Trove connection reset while updating the status to "success". ` sqlalchemy.e... [06:50:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [07:04:00] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#9891544 (10Slst2020) >>! In T352417#9815607, @Slst2020 wrote: > Robot accounts still don't have update permissions on project quotas. A... [07:05:43] (03update) 10sstefanova: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (owner: 10raymond-ndibe) [07:05:50] (03merge) 10sstefanova: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (owner: 10raymond-ndibe) [07:05:53] (03update) 10raymond-ndibe: api: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/92 (owner: 10sstefanova) [07:08:35] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.307-20240614070606-03ccd024 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/333 (https://phabricator.wikimedia.org/T356974) [07:11:23] 10Quarry, 13Patch-Needs-Improvement: EXPLAIN is broken because new analytics wiki replica cluster contains multiple servers - https://phabricator.wikimedia.org/T205214#9891551 (10SD0001) [07:11:53] 10Quarry: quarry explain not working since move to multiple databases - https://phabricator.wikimedia.org/T288170#9891549 (10SD0001) →14Duplicate dup:03T205214 [07:15:07] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9891554 (10SD0001) p:05Triage→03High [07:21:47] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [07:21:58] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [07:24:39] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9891576 (10github-toolforge-bot) siddharthvp opened https://github.com/toolforge/quarry/pull/51 [07:24:41] siddharthvp opened https://github.com/toolforge/quarry/pull/51 [07:25:28] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [07:33:06] 10Cloud-VPS, 10Quarry: [bug] Lot of queries stuck in queued state for hours and days - https://phabricator.wikimedia.org/T365136#9891580 (10SD0001) The actual problem should be sorted with the fix for T367464. [07:35:01] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [07:35:12] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [07:40:44] 10Quarry: query runs forever - https://phabricator.wikimedia.org/T366909#9891592 (10SD0001) Or more likely because of a connection reset from Trove (T367464). [07:40:51] 10Quarry: query runs forever - https://phabricator.wikimedia.org/T366909#9891597 (10SD0001) →14Duplicate dup:03T367464 [07:41:07] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9891599 (10SD0001) [07:43:44] (03update) 10sstefanova: jobs-api: bump to 0.0.307-20240614070606-03ccd024 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/333 (https://phabricator.wikimedia.org/T356974) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [07:44:23] (03merge) 10sstefanova: jobs-api: bump to 0.0.307-20240614070606-03ccd024 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/333 (https://phabricator.wikimedia.org/T356974) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [07:50:55] (03merge) 10sstefanova: api: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/92 [07:53:30] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.308-20240614075107-8b4683d2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/334 [07:55:26] 10Tool-containers, 10Toolforge: Provide a Redis container for use within a tool's namespace - https://phabricator.wikimedia.org/T360378#9891643 (10Pintoch) Ok, I wonder if it could be linked to the fact that I tried creating the `redis` job earlier, back when the quota didn't allow for it? I'm not sure. [08:02:20] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [08:02:30] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [08:09:39] 10Tools: Quickstatements failing to OAuth - https://phabricator.wikimedia.org/T367430#9891663 (10Aklapper) 05Open→03Invalid Please report this in the issuetracker of QuickStatements at https://github.com/magnusmanske/quickstatements/issues as the maintainer(s) do(es) not use Wikimedia Phabricator for pla... [08:14:54] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [08:15:05] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [08:17:24] 10Tool-citationanalyzer: Citation Analyzer is currently unavailable - https://phabricator.wikimedia.org/T367489#9891675 (10Aklapper) [08:18:02] 10Tool-citationanalyzer: Citation Analyzer is currently unavailable - https://phabricator.wikimedia.org/T367489#9891676 (10Aklapper) Hi @Baswameenakshi, thanks for taking the time to report this and welcome to Wikimedia Phabricator! Have you used that tool before? How did you end up trying to use this particular... [08:25:29] (03update) 10sstefanova: jobs-api: bump to 0.0.308-20240614075107-8b4683d2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/334 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:25:38] (03approved) 10sstefanova: jobs-api: bump to 0.0.308-20240614075107-8b4683d2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/334 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:25:45] (03merge) 10sstefanova: jobs-api: bump to 0.0.308-20240614075107-8b4683d2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/334 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:26:11] (03update) 10sstefanova: jobs-api: bump to 0.0.307-20240614070606-03ccd024 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/333 (https://phabricator.wikimedia.org/T356974) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:28:19] (03merge) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/38 [09:06:49] (03open) 10sstefanova: d/changelog: bump to 16.0.11 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/39 (https://phabricator.wikimedia.org/T366674) [09:10:40] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9891772 (10fnegri) @Liz are the queries that never finish different from other queries, or are they similar but sometimes they randomly f... [09:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:49:34] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9891856 (10github-toolforge-bot) siddharthvp closed https://github.com/toolforge/quarry/pull/51 [09:49:41] siddharthvp closed https://github.com/toolforge/quarry/pull/51 [10:05:10] (03close) 10sstefanova: d/changelog: bump to 16.0.11 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/39 (https://phabricator.wikimedia.org/T366674) [10:12:56] 10Quarry: refreshing a running query changes favicon from orange to blue - https://phabricator.wikimedia.org/T362101#9891899 (10github-toolforge-bot) siddharthvp opened https://github.com/toolforge/quarry/pull/52 [10:13:05] siddharthvp opened https://github.com/toolforge/quarry/pull/52 [10:18:45] (03open) 10sstefanova: cli: remove deprecated endpoint calls [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/40 [10:18:47] (03update) 10sstefanova: cli: remove deprecated endpoint calls [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/40 [10:24:20] (03update) 10sstefanova: cli: remove deprecated endpoint calls [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/40 [10:28:35] (03approved) 10aborrero: cli: remove deprecated endpoint calls [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/40 (owner: 10sstefanova) [10:31:53] (03update) 10sstefanova: cli: remove deprecated endpoint calls [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/40 [10:32:18] (03merge) 10sstefanova: cli: remove deprecated endpoint calls [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/40 [10:39:47] 10Toolforge (Toolforge iteration 11): [maintain-kubeusers,infra] deal with tools with invalid names - https://phabricator.wikimedia.org/T366477#9891938 (10taavi) 05Open→03Resolved [10:50:07] 10Quarry, 13Patch-Needs-Improvement: EXPLAIN is broken because new analytics wiki replica cluster contains multiple servers - https://phabricator.wikimedia.org/T205214#9891953 (10github-toolforge-bot) siddharthvp opened https://github.com/toolforge/quarry/pull/53 [10:50:10] siddharthvp opened https://github.com/toolforge/quarry/pull/53 [11:04:12] (03open) 10sstefanova: d/changelog: bump to 16.0.11 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/41 (https://phabricator.wikimedia.org/T366674) [11:10:43] 10Quarry: refreshing a running query changes favicon from orange to blue - https://phabricator.wikimedia.org/T362101#9891976 (10github-toolforge-bot) siddharthvp closed https://github.com/toolforge/quarry/pull/52 [11:10:48] siddharthvp closed https://github.com/toolforge/quarry/pull/52 [11:45:00] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9892038 (10Ladsgroup) I will try to fix it during this weekend (to also minimize disruptions) [12:00:46] (03PS1) 10Majavah: openstack: cloudnet: Remove migrate_to_ovs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043743 [12:00:46] (03PS1) 10Majavah: openstack: Add cookbook to migrate a server to OVS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043744 (https://phabricator.wikimedia.org/T326373) [12:03:44] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server taavi-hostkeys [12:03:44] (03CR) 10CI reject: [V:04-1] openstack: Add cookbook to migrate a server to OVS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043744 (https://phabricator.wikimedia.org/T326373) (owner: 10Majavah) [12:03:50] !log taavi@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server taavi-hostkeys [12:05:34] (03PS2) 10Majavah: openstack: Add cookbook to migrate a server to OVS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043744 (https://phabricator.wikimedia.org/T326373) [12:05:42] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server taavi-hostkeys [12:06:20] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9892077 (10taavi) [12:06:47] !log taavi@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server taavi-hostkeys [12:10:17] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server taavi-vagrant [12:10:18] !log taavi@cloudcumin1001 testlabs END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server taavi-vagrant [12:10:24] (03PS3) 10Majavah: openstack: Add cookbook to migrate a server to OVS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043744 (https://phabricator.wikimedia.org/T326373) [12:10:29] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server taavi-vagrant [12:11:31] !log taavi@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server taavi-vagrant [12:15:32] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 87975ac8-d625-44e7-a7db-c150d4430f45 [12:16:25] !log taavi@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 87975ac8-d625-44e7-a7db-c150d4430f45 [12:16:34] (03PS4) 10Majavah: openstack: Add cookbook to migrate a server to OVS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043744 (https://phabricator.wikimedia.org/T326373) [12:19:55] (03CR) 10CI reject: [V:04-1] openstack: Add cookbook to migrate a server to OVS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043744 (https://phabricator.wikimedia.org/T326373) (owner: 10Majavah) [12:21:07] (03PS5) 10Majavah: openstack: Add cookbook to migrate a server to OVS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043744 (https://phabricator.wikimedia.org/T326373) [12:22:20] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server taavi-loki-1 [12:23:20] !log taavi@cloudcumin1001 testlabs END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server taavi-loki-1 [12:23:49] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server taavi-testpuppet-1 [12:24:59] !log taavi@cloudcumin1001 testlabs END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server taavi-testpuppet-1 [12:29:04] (03update) 10raymond-ndibe: [envvars-cli] remove unused code [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/41 [12:38:21] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 06Infrastructure-Foundations, 10Observability-Alerting, 05Goal: [wmcs-cookbooks] Downtime alerts from cloudcumins - https://phabricator.wikimedia.org/T347490#9892235 (10taavi) 05Open→03Resolved [12:58:06] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server taavi-puppet7-server [12:58:59] !log taavi@cloudcumin1001 testlabs END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server taavi-puppet7-server [12:59:35] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server taavi-puppet7-server [12:59:38] !log taavi@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server taavi-puppet7-server [13:01:48] !log taavi@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_server_to_ovs for server taavi-xmltest [13:02:48] !log taavi@cloudcumin1001 testlabs END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server taavi-xmltest [13:03:17] 10Cloud-VPS, 10Quarry: [bug] Lot of queries stuck in queued state for hours and days - https://phabricator.wikimedia.org/T365136#9892340 (10SD0001) 05Open→03Resolved a:03SD0001 The fix is live. Please reopen if this occurs again. [13:04:04] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9892343 (10SD0001) 05Open→03Resolved a:03SD0001 The fix is live. Please reopen if the problem arises again. [13:04:09] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' [13:07:43] (03PS6) 10Majavah: openstack: Add cookbook to migrate a server to OVS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043744 (https://phabricator.wikimedia.org/T326373) [13:08:00] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1 [13:09:01] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1 [13:10:25] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05 [13:11:14] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05 [13:11:40] (03update) 10raymond-ndibe: [envvar-api] remove unused code [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/30 [13:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:18:03] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' [13:21:53] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9892402 (10taavi) [13:44:16] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:47:38] (03PS2) 10Hashar: Archive repository [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/1040858 (https://phabricator.wikimedia.org/T269703) (owner: 10Pppery) [13:47:47] (03CR) 10CI reject: [V:04-1] Archive repository [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/1040858 (https://phabricator.wikimedia.org/T269703) (owner: 10Pppery) [13:48:22] (03CR) 10Hashar: [V:03+2 C:03+2] Archive repository [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/1040858 (https://phabricator.wikimedia.org/T269703) (owner: 10Pppery) [13:48:28] (03CR) 10CI reject: [V:04-1] Archive repository [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/1040858 (https://phabricator.wikimedia.org/T269703) (owner: 10Pppery) [13:49:16] RESOLVED: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:49:45] (03Abandoned) 10Hashar: Added a canary to the coal mine [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/220294 (owner: 10Sitic) [13:49:48] (03Abandoned) 10Hashar: Add PLURAL support for SHOW_LAST_DAYS message [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265804 (owner: 10MtDu) [13:49:51] (03Abandoned) 10Hashar: Add PLURAL support to LOGEVENT_RENAMEUSER message [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265647 (https://phabricator.wikimedia.org/T114876) (owner: 10MtDu) [14:11:21] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [14:11:31] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [14:12:37] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9892642 (10taavi) [14:31:34] 10Quarry: Add line numbers in SQL input textarea - https://phabricator.wikimedia.org/T315066#9892702 (10SD0001) 05Open→03In progress [14:35:33] (03open) 10taavi: jobs-api: Rollback tools to 0.0.305 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/335 [14:36:45] 10Toolforge: `toolforge jobs load jobs.yaml` crashes - https://phabricator.wikimedia.org/T367520 (10Soda) 03NEW [14:37:30] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 13Patch-For-Review: Remove namespace 666 from Wikitech - https://phabricator.wikimedia.org/T367254#9892765 (10taavi) That namespace does have deleted pages, can MW handle deleting a namespace with those in there? [14:37:41] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 13Patch-For-Review: Remove namespace 666 from Wikitech - https://phabricator.wikimedia.org/T367254#9892766 (10taavi) p:05Triage→03Low [14:37:53] (03update) 10taavi: jobs-api: Rollback tools to 0.0.305 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/335 (https://phabricator.wikimedia.org/T367520) [14:37:54] (03update) 10taavi: jobs-api: Rollback tools to 0.0.305 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/335 (https://phabricator.wikimedia.org/T367520) [14:38:37] (03merge) 10taavi: jobs-api: Rollback tools to 0.0.305 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/335 (https://phabricator.wikimedia.org/T367520) [14:38:41] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [14:38:51] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [14:41:19] 10Quarry, 13Patch-Needs-Improvement: EXPLAIN is broken because new analytics wiki replica cluster contains multiple servers - https://phabricator.wikimedia.org/T205214#9892773 (10github-toolforge-bot) siddharthvp closed https://github.com/toolforge/quarry/pull/53 [14:41:28] siddharthvp closed https://github.com/toolforge/quarry/pull/53 [14:43:24] 10Toolforge: `toolforge jobs load jobs.yaml` crashes - https://phabricator.wikimedia.org/T367520#9892778 (10taavi) p:05Triage→03High I rolled back today's `jobs-api` deployments. This includes: ` 8b4683d 4 days ago api: prefix endpoints with /tool/{toolname}/ [Slavina Stefanova] [Slavina Stefanova] 03ccd02 1... [14:44:04] 10Quarry: Add line numbers in SQL input textarea - https://phabricator.wikimedia.org/T315066#9892781 (10github-toolforge-bot) siddharthvp opened https://github.com/toolforge/quarry/pull/54 [14:44:13] siddharthvp opened https://github.com/toolforge/quarry/pull/54 [14:44:55] 10Toolforge: `toolforge jobs load jobs.yaml` crashes - https://phabricator.wikimedia.org/T367520#9892795 (10Soda) [14:45:35] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "auditlogging" project Buster deprecation - https://phabricator.wikimedia.org/T367522 (10StrikerBot) 03NEW p:05Triage→03Medium [14:45:41] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "clouddb-services" project Buster deprecation - https://phabricator.wikimedia.org/T367523 (10StrikerBot) 03NEW p:05Triage→03Medium [14:45:47] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "codesearch" project Buster deprecation - https://phabricator.wikimedia.org/T367524 (10StrikerBot) 03NEW p:05Triage→03Medium [14:45:50] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "commons-corruption-checker" project Buster deprecation - https://phabricator.wikimedia.org/T367525 (10StrikerBot) 03NEW p:05Triage→03Medium [14:45:55] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "dashiki" project Buster deprecation - https://phabricator.wikimedia.org/T367526 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:28] (03open) 10taavi: Revert "jobs-api: Rollback tools to 0.0.305" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/336 [14:47:30] (03merge) 10taavi: Revert "jobs-api: Rollback tools to 0.0.305" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/336 [14:47:30] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "download" project Buster deprecation - https://phabricator.wikimedia.org/T367527 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:31] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "dumps" project Buster deprecation - https://phabricator.wikimedia.org/T367528 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:31] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [14:47:33] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "etytree" project Buster deprecation - https://phabricator.wikimedia.org/T367529 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:35] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "eventmetrics" project Buster deprecation - https://phabricator.wikimedia.org/T367530 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:38] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "globaleducation" project Buster deprecation - https://phabricator.wikimedia.org/T367531 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:42] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [14:47:42] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "google-api-proxy" project Buster deprecation - https://phabricator.wikimedia.org/T367532 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:47] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "image-suggestion-api" project Buster deprecation - https://phabricator.wikimedia.org/T367533 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:51] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "linkwatcher" project Buster deprecation - https://phabricator.wikimedia.org/T367536 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:55] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "k8splay" project Buster deprecation - https://phabricator.wikimedia.org/T367535 (10StrikerBot) 03NEW p:05Triage→03Medium [14:47:59] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "integration" project Buster deprecation - https://phabricator.wikimedia.org/T367534 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:03] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "machine-learning" project Buster deprecation - https://phabricator.wikimedia.org/T367537 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:07] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mailman" project Buster deprecation - https://phabricator.wikimedia.org/T367538 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:11] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "maps-experiments" project Buster deprecation - https://phabricator.wikimedia.org/T367539 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:15] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "math" project Buster deprecation - https://phabricator.wikimedia.org/T367540 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:19] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mediawiki-vagrant" project Buster deprecation - https://phabricator.wikimedia.org/T367541 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:25] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9892912 (10Liz) I don't see that there has been any improvement. My queries are still not finishing their run. [14:48:29] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mwv-apt" project Buster deprecation - https://phabricator.wikimedia.org/T367542 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:33] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "osmit" project Buster deprecation - https://phabricator.wikimedia.org/T367543 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:38] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "packaging" project Buster deprecation - https://phabricator.wikimedia.org/T367544 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:42] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "petscan" project Buster deprecation - https://phabricator.wikimedia.org/T367545 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:50] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "pki" project Buster deprecation - https://phabricator.wikimedia.org/T367546 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:54] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "puppet-diffs" project Buster deprecation - https://phabricator.wikimedia.org/T367547 (10StrikerBot) 03NEW p:05Triage→03Medium [14:48:58] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "rcm" project Buster deprecation - https://phabricator.wikimedia.org/T367548 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:02] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "recommendation-api" project Buster deprecation - https://phabricator.wikimedia.org/T367549 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:08] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:12] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "research-collaborations-api" project Buster deprecation - https://phabricator.wikimedia.org/T367551 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:16] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "schematreerecommender" project Buster deprecation - https://phabricator.wikimedia.org/T367552 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:20] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "shiny-r" project Buster deprecation - https://phabricator.wikimedia.org/T367553 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:30] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "sso" project Buster deprecation - https://phabricator.wikimedia.org/T367554 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:34] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "striker" project Buster deprecation - https://phabricator.wikimedia.org/T367555 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:38] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "toolhub" project Buster deprecation - https://phabricator.wikimedia.org/T367556 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:42] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "toolsbeta" project Buster deprecation - https://phabricator.wikimedia.org/T367557 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:46] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "videocuttool" project Buster deprecation - https://phabricator.wikimedia.org/T367558 (10StrikerBot) 03NEW p:05Triage→03Medium [14:49:55] 10Toolforge: `toolforge jobs load jobs.yaml` crashes - https://phabricator.wikimedia.org/T367520#9892993 (10taavi) 05Open→03Invalid Oh, no, nevermind, I too was testing with an invalid file. I'll file a separate bug for the validation issue. [14:49:59] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikidumpparse" project Buster deprecation - https://phabricator.wikimedia.org/T367561 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:03] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikicommunityhealth" project Buster deprecation - https://phabricator.wikimedia.org/T367560 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:07] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "visualeditor" project Buster deprecation - https://phabricator.wikimedia.org/T367559 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:11] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikilabels" project Buster deprecation - https://phabricator.wikimedia.org/T367562 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:15] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikipathways" project Buster deprecation - https://phabricator.wikimedia.org/T367563 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:19] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikisource" project Buster deprecation - https://phabricator.wikimedia.org/T367564 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:23] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikispeech" project Buster deprecation - https://phabricator.wikimedia.org/T367565 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:29] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikispore" project Buster deprecation - https://phabricator.wikimedia.org/T367566 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:33] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wm-bot" project Buster deprecation - https://phabricator.wikimedia.org/T367567 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:37] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wmde-dashboards" project Buster deprecation - https://phabricator.wikimedia.org/T367568 (10StrikerBot) 03NEW p:05Triage→03Medium [14:50:43] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9893183 (10Liz) 05Resolved→03Open [14:51:21] 10Toolforge: toolforge jobs load crashes if given YAML is an object (instead of an array) - https://phabricator.wikimedia.org/T367569 (10taavi) 03NEW [14:52:02] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "toolsbeta" project Buster deprecation - https://phabricator.wikimedia.org/T367557#9893232 (10taavi) →14Duplicate dup:03T360699 [14:52:08] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Toolsbeta: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T360699#9893246 (10taavi) [14:52:28] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "codesearch" project Buster deprecation - https://phabricator.wikimedia.org/T367524#9893240 (10Andrew) →14Duplicate dup:03T367479 [14:52:32] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9893242 (10Andrew) [14:53:04] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "clouddb-services" project Buster deprecation - https://phabricator.wikimedia.org/T367523#9893262 (10taavi) [14:53:08] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-Projects, 10Puppet (Puppet 7.0): Are clouddb-wikireplicas-query-1 and the cloudb-services project still useful? - https://phabricator.wikimedia.org/T359810#9893263 (10taavi) [14:53:16] 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations, 10Puppet CI: Cloud VPS "puppet-diffs" project Buster deprecation - https://phabricator.wikimedia.org/T367547#9893273 (10taavi) [14:53:25] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Toolsbeta: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T360699#9893234 (10taavi) [14:59:18] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 13Patch-For-Review: Remove namespace 666 from Wikitech - https://phabricator.wikimedia.org/T367254#9893302 (10DreamRimmer) @taavi, I believe MW can handle this for custom namespaces, but since it involves cleaning up the archive table, I think a sysadmin s... [15:00:08] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikispore" project Buster deprecation - https://phabricator.wikimedia.org/T367566#9893303 (10Tgr) a:03Tgr We expect this to be done on time. [15:01:11] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "videocuttool" project Buster deprecation - https://phabricator.wikimedia.org/T367558#9893320 (10Soda) I haven't had time to look at Buster migration, but will do it within 15 July :) [15:01:13] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispore: Rebuild Wikispore Vagrant boxes on Bullseye or Bookworm - https://phabricator.wikimedia.org/T365934#9893321 (10Tgr) [15:01:31] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikispore" project Buster deprecation - https://phabricator.wikimedia.org/T367566#9893309 (10Tgr) →14Duplicate dup:03T365934 [15:02:11] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispore: Rebuild Wikispore Vagrant boxes on Bullseye or Bookworm - https://phabricator.wikimedia.org/T365934#9893311 (10Tgr) [15:02:14] 10Cloud-VPS (Debian Buster Deprecation), 06Editing-team: Cloud VPS "visualeditor" project Buster deprecation - https://phabricator.wikimedia.org/T367559#9893326 (10matmarex) Excusing myself and the folks who were added to the project to have access to Patch demo (which is in this project for historical reasons... [15:06:01] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mediawiki-vagrant" project Buster deprecation - https://phabricator.wikimedia.org/T367541#9893336 (10Tgr) Note this project is [[https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2024_Purge#mediawiki-vagrant|unclaimed]]. Unlike mwv-apt (T367542) it's not used... [15:07:25] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mwv-apt" project Buster deprecation - https://phabricator.wikimedia.org/T367542#9893342 (10Tgr) a:03Tgr I'll look into this. IIRC mwv-apt-02 is uses as an APT source by Vagrant; not sure how useful/important it is. [15:07:33] 10Cloud-VPS (Debian Buster Deprecation), 06Release-Engineering-Team: Cloud VPS "integration" project Buster deprecation - https://phabricator.wikimedia.org/T367534#9893362 (10taavi) [15:11:37] 10Toolforge: toolforge jobs load crashes if given YAML is an object (instead of an array) - https://phabricator.wikimedia.org/T367569#9893393 (10bd808) [15:22:09] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "recommendation-api" project Buster deprecation - https://phabricator.wikimedia.org/T367549#9893440 (10Isaac) @Andrew I'd like to understand our options a bit and also tagging @Pginer-WMF and @Nikerabbit so you're aware and can chime in. The `tool` instance in q... [15:38:11] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "research-collaborations-api" project Buster deprecation - https://phabricator.wikimedia.org/T367551#9893480 (10Isaac) @diego so you're aware re: fact-checking endpoint. @MunizaA thoughts on this migration for wikinav? I'm happy to take a try at it but would n... [15:40:20] 10Cloud-VPS, 10Quarry: [bug] Lot of queries stuck in queued state for hours and days - https://phabricator.wikimedia.org/T365136#9893481 (10Teslaton) @SD0001: would it be possible to batch fail queries affected by this problem in latest months - i.e. to mark them as failed instead of current "queued" state... [15:40:44] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 13Patch-For-Review: Remove namespace 666 from Wikitech - https://phabricator.wikimedia.org/T367254#9893485 (10Bugreporter) Previously we did remove namespaces multiple times, e.g. T354813 and T285766. [15:42:00] 10Cloud-VPS, 10Quarry: [bug] Lot of queries stuck in queued state for hours and days - https://phabricator.wikimedia.org/T365136#9893490 (10Novem_Linguae) Usually if I press stop and start a couple times on my own queries, they will restart. They are also forkable. [15:44:36] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "k8splay" project Buster deprecation - https://phabricator.wikimedia.org/T367535#9893498 (10Dzahn) I will take care of dzahn.k8splay.eqiad1.wikimedia.cloud. @ArielGlenn atgs-playground sounds like yours. [15:44:57] 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations: Cloud VPS "packaging" project Buster deprecation - https://phabricator.wikimedia.org/T367544#9893499 (10Dzahn) [15:45:54] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9893503 (10Dzahn) I probably shouldn't be admin of this project. I hardly remember why I am in it. Years ago helped fixing something I guess. [15:46:33] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9893504 (10Dzahn) I can help. [15:47:06] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: server fails to reboot for clouddb1018.eqiad.wmnet - https://phabricator.wikimedia.org/T367499#9893507 (10fnegri) [15:53:48] 10Toolforge: `toolforge jobs load jobs.yaml` crashes - https://phabricator.wikimedia.org/T367520#9893540 (10Slst2020) @taavi thanks for the rollback. I'll take a look on Monday. The functional tests all passed, so probably we will need some additional ones to catch this. [16:02:35] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9893592 (10bd808) >>! In T367550#9893503, @Dzahn wrote: > I probably shouldn't be admin of this project. I hardly remember why I am in it. Years ago helped fixing something I gues... [16:03:45] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9893596 (10bd808) a:03bd808 [16:11:32] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mediawiki-vagrant" project Buster deprecation - https://phabricator.wikimedia.org/T367541#9893624 (10bd808) We used to use this project to produce an iso image for installing MediaWiki-Vagrant at Hackathons and LAN parties. https://mwv.wmcloud.org/ is very obvi... [16:14:27] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "toolhub" project Buster deprecation - https://phabricator.wikimedia.org/T367556#9893642 (10bd808) a:03bd808 [16:15:27] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "download" project Buster deprecation - https://phabricator.wikimedia.org/T367527#9893652 (10bd808) I think @Andrew was looking into sunsetting this project recently. [16:23:33] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "striker" project Buster deprecation - https://phabricator.wikimedia.org/T367555#9893690 (10bd808) It is probably past time for #cloud-services-team to take over this project along with #striker in general as #toolforge infrastructure. What are your thoughts on... [16:40:45] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "k8splay" project Buster deprecation - https://phabricator.wikimedia.org/T367535#9893754 (10Dzahn) [16:49:39] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9893780 (10Dzahn) Hmm.. my bash history shows I ran puppet agent twice and that was it. I see these: ` server { listen 80; server_name phab.wmcloud.org; return 301 https:... [16:57:19] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9893860 (10Dzahn) created new instance `codesearch9` (g4.cores4.ram8.disk20) to replace `codesearch8` (g3... [17:01:21] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9893871 (10Ladsgroup) >>! In T367479#9893504, @Dzahn wrote: > I can help. Thank you <3 I will pick up fr... [17:08:30] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wm-bot" project Buster deprecation - https://phabricator.wikimedia.org/T367567#9893896 (10MacFan4000) I already moved it to new VMs a while ago. The old ones are shutoff and aren’t needed anymore. [17:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:14:51] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wmde-dashboards" project Buster deprecation - https://phabricator.wikimedia.org/T367568#9893914 (10AndrewTavis_WMDE) Thanks for making this! I've marked the `wmde-dashboards` project as good for deletion as seen [here](https://wikitech.wikimedia.org/wiki/News/C... [17:22:14] 10Tool-containers, 10Toolforge: Provide a Redis container for use within a tool's namespace - https://phabricator.wikimedia.org/T360378#9893932 (10bd808) >>! In T360378#9891643, @Pintoch wrote: > Ok, I wonder if it could be linked to the fact that I tried creating the `redis` job earlier, back when the quota d... [17:25:13] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wmde-dashboards" project Buster deprecation - https://phabricator.wikimedia.org/T367568#9893937 (10JJMC89) 05Open→03Declined [17:31:23] 06cloud-services-team, 10Cloud-VPS: [openstack] APT failing to update osbpo packages in Cloud instances - https://phabricator.wikimedia.org/T366028#9893965 (10Andrew) 05Open→03Resolved a:03Andrew I've build a new bullseye base image, and forced puppet to rebuild the sources entry for existing VMs. So... [17:32:22] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Launching new bullseye deployment-prep instances fails, no sudo access - https://phabricator.wikimedia.org/T361536#9893971 (10Andrew) 05Open→03Resolved I have built several bullseye VMs without trouble so I not believe this to be f... [17:37:56] FIRING: [2x] SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:42:36] (03CR) 10Andrew Bogott: ensure_canary: use new g4 flavor for canary instances (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043149 (owner: 10Andrew Bogott) [17:44:05] (03PS2) 10Andrew Bogott: ensure_canary: use new g4 flavor for canary instances [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043149 [17:44:18] (03CR) 10Andrew Bogott: ensure_canary: use new g4 flavor for canary instances (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043149 (owner: 10Andrew Bogott) [17:52:40] 10Tool-wd-image-positions: Logging in causes the user to go back to the index page - https://phabricator.wikimedia.org/T367188#9894037 (10LucasWerkmeister) 05Open→03Resolved a:03LucasWerkmeister Seems to work now :) [18:04:59] (03CR) 10Majavah: ensure_canary: use new g4 flavor for canary instances (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043149 (owner: 10Andrew Bogott) [18:12:56] FIRING: [2x] SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:17:45] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Research: Replace or remove Debian Buster VMs in 'wmf-research-tools' cloud-vps project - https://phabricator.wikimedia.org/T367444#9894134 (10taavi) a:05Andrew→03None [18:32:49] 10Quarry: Deduplicate config load - https://phabricator.wikimedia.org/T349135#9894202 (10github-toolforge-bot) siddharthvp opened https://github.com/toolforge/quarry/pull/55 [18:32:55] siddharthvp opened https://github.com/toolforge/quarry/pull/55 [18:32:56] RESOLVED: SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudservices1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:51:59] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "download" project Buster deprecation - https://phabricator.wikimedia.org/T367527#9894270 (10Andrew) I variety of clients are downloading things under 'corpora' pretty frequently. So perhaps killing this project this should wait until we have a proper toolforge... [18:59:30] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: server fails to reboot for clouddb1018.eqiad.wmnet - https://phabricator.wikimedia.org/T367499#9894294 (10Marostegui) Sometimes it means the host is stuck at a memory check - should be visible onsite. [19:02:10] FIRING: [2x] ProjectProxyMainProxyDown: Proxy on proxy-03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown [19:04:54] 06cloud-services-team: Replace 'download' cloud-vps project afater we support per-tool object storage - https://phabricator.wikimedia.org/T367593 (10Andrew) 03NEW [19:05:40] 06cloud-services-team: Replace 'download' cloud-vps project afater we support per-tool object storage - https://phabricator.wikimedia.org/T367593#9894318 (10Andrew) [19:05:44] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9894319 (10Andrew) [19:05:49] 06cloud-services-team: Replace 'download' cloud-vps project after we support per-tool object storage - https://phabricator.wikimedia.org/T367593#9894320 (10Andrew) [19:07:10] RESOLVED: [2x] ProjectProxyMainProxyDown: Proxy on proxy-03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown [19:07:46] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "download" project Buster deprecation - https://phabricator.wikimedia.org/T367527#9894321 (10Andrew) 05Open→03Resolved a:03Andrew I've created T367593 about shutting down this project, and replaced the Buster download server with one running Bookworm. [19:12:57] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-Projects, 10Puppet (Puppet 7.0): Are clouddb-wikireplicas-query-1 and the cloudb-services project still useful? - https://phabricator.wikimedia.org/T359810#9894332 (10Andrew) 05Open→03Resolved a:03Andrew deleted! [19:15:28] FIRING: InstanceDown: Project clouddb-services instance clouddb-services-puppetserver-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:17:56] FIRING: [2x] SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:22:56] FIRING: [3x] SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:30:43] FIRING: [2x] InstanceDown: Project clouddb-services instance clouddb-services-puppetserver-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:32:05] 10Cloud-VPS (Debian Buster Deprecation), 06The-Wikipedia-Library, 10Moderator-Tools-Team (Kanban): Replace deprecated Buster VMs in Cloud VPS - https://phabricator.wikimedia.org/T364399#9894374 (10jsn.sherman) [19:32:56] RESOLVED: [3x] SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:34:13] 10Quarry: Show replication lag - https://phabricator.wikimedia.org/T60841#9894368 (10SD0001) 05Open→03Resolved a:03Framawiki https://github.com/toolforge/quarry/pull/22/ was merged last year. [19:36:14] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispeech: Cloud VPS "wikispeech" project Buster deprecation - https://phabricator.wikimedia.org/T367565#9894376 (10Pppery) [19:38:16] 10Cloud-VPS (Debian Buster Deprecation), 06Machine-Learning-Team, 10Wikilabels: Cloud VPS "wikilabels" project Buster deprecation - https://phabricator.wikimedia.org/T367562#9894377 (10Pppery) [19:38:19] 10Cloud-VPS (Debian Buster Deprecation), 10VideoCutTool: Cloud VPS "videocuttool" project Buster deprecation - https://phabricator.wikimedia.org/T367558#9894390 (10Pppery) [19:43:26] 10Cloud-VPS (Debian Buster Deprecation), 06Community-Tech, 10Google-api-proxy: Cloud VPS "google-api-proxy" project Buster deprecation - https://phabricator.wikimedia.org/T367532#9894408 (10Pppery) [20:00:07] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "auditlogging" project Buster deprecation - https://phabricator.wikimedia.org/T367522#9894459 (10Southparkfan) Relevant: T127717#9671526 (i.e. can be deleted without issues, but preferably only before other instances are deleted) [20:39:15] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9894552 (10Dzahn) ah, we had another issue on the existing machine as well.. the puppet 7 migration: ` Notice: puppet7 is not available on buster. forcing this is likely going... [20:50:23] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9894566 (10Dzahn) @bd808 woah, the puppet role just works on bookworm.. not even a warning! 🤯 ` dzahn@redirects-nginx03:~$ for host in $(grep server_name /etc/nginx/sites-enab... [20:51:44] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "eventmetrics" project Buster deprecation - https://phabricator.wikimedia.org/T367530#9894570 (10MusikAnimal) I have this nearly done, but it's not done yet (the forced PHP upgrade meant I had to rewrite a bunch of things). #community-tech has a higher priority... [20:54:42] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9894579 (10Dzahn) ` 16:54 <@bd808> mutante: yeah, that should pretty much be all it takes. Setup a instance with the role, do some localhost curl tests to see that it is emitting... [21:00:53] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "image-suggestion-api" project Buster deprecation - https://phabricator.wikimedia.org/T367533#9894625 (10BPirkle) I've deleted everything related to this that I could find. If there are more pieces left, feel free to delete them. [21:01:00] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "image-suggestion-api" project Buster deprecation - https://phabricator.wikimedia.org/T367533#9894626 (10BPirkle) [21:11:37] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9894652 (10Dzahn) Everything works from localhost and also from old host to IP of new host. But in my browser phab.wmcloud.org and phorge.wmcloud.org timeout. So looks like the... [21:11:52] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9894653 (10Dzahn) 05Open→03In progress [21:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 13 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:14:26] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9894658 (10Dzahn) This is a good opportunity because it will be a good chunk of T268199 (moving codesearc... [21:15:44] 10VPS-project-Codesearch, 06collaboration-services: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#9894661 (10Dzahn) Now we have to upgrade the buster machines in cloud anyways, for T367479. This is a good opportunity because it's basically a good chunk of the work also needed f... [21:16:39] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550#9894665 (10Dzahn) old IP: 172.16.0.48 new IP: 172.16.7.2 port 80 [21:31:40] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9894686 (10Dzahn) Like with other buster machines we also have the buster/puppet7 conflict here: ` Noti... [21:41:11] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9894699 (10Dzahn) Things to fix: ` E: Package 'docker-ce' has no installation candidate .. Execution of... [21:55:34] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-Projects, 10Puppet (Puppet 7.0): Are clouddb-wikireplicas-query-1 and the cloudb-services project still useful? - https://phabricator.wikimedia.org/T359810#9894732 (10fnegri) I didn't see this task, but I had left a comment at https:... [22:04:08] 10Data-Services: [cloud-vps] Deprecate clouddb-services project - https://phabricator.wikimedia.org/T365975#9894763 (10fnegri) [22:04:09] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-Projects, 10Puppet (Puppet 7.0): Are clouddb-wikireplicas-query-1 and the cloudb-services project still useful? - https://phabricator.wikimedia.org/T359810#9894764 (10fnegri) [22:04:19] 10Data-Services: [cloud-vps] Deprecate clouddb-services project - https://phabricator.wikimedia.org/T365975#9894760 (10fnegri) The project was deleted today (T359810) but the DNS names are still listed as belonging to that project: ` fnegri@cloudcontrol1005:~$ sudo wmcs-openstack zone list --all-projects |grep... [22:06:49] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "clouddb-services" project Buster deprecation - https://phabricator.wikimedia.org/T367523#9894771 (10fnegri) [22:07:02] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "clouddb-services" project Buster deprecation - https://phabricator.wikimedia.org/T367523#9894772 (10fnegri) [22:07:31] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "clouddb-services" project Buster deprecation - https://phabricator.wikimedia.org/T367523#9894766 (10fnegri) 05Open→03Resolved a:03fnegri The only instance in this project was deleted in T359810. [22:07:53] 10Data-Services: [cloud-vps] migrate DNS zones away from deprecated clouddb-services project - https://phabricator.wikimedia.org/T365975#9894775 (10fnegri) [22:25:51] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch, 13Patch-For-Review: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479#9894943 (10Dzahn) @Ladsgroup So, docker-ce won't be found on bookworm. There is doc... [22:56:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-nfs-1 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:04:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:11:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance cvn-nfs-1 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:11:43] 10Data-Services: [cloud-vps] migrate DNS zones away from deprecated clouddb-services project - https://phabricator.wikimedia.org/T365975#9894995 (10fnegri) There are actually 7 zones in total in the project: ` fnegri@cloudcontrol1005:~$ sudo wmcs-openstack zone list --all-projects |grep clouddb-services | 4b927... [23:17:32] 10Data-Services: [cloud-vps] migrate DNS zones away from deprecated clouddb-services project - https://phabricator.wikimedia.org/T365975#9895006 (10fnegri) But the extra ones are just standard openstack zones that are not being used. @Andrew just migrated the 3 ones with useful records to the `cloudinfra` proje... [23:21:49] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-Projects, 10Puppet (Puppet 7.0): Are clouddb-wikireplicas-query-1 and the cloudb-services project still useful? - https://phabricator.wikimedia.org/T359810#9895016 (10Andrew) >>! In T359810#9894732, @fnegri wrote: > I didn't see this... [23:24:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:24:55] 10Data-Services: [cloud-vps] migrate DNS zones away from deprecated clouddb-services project - https://phabricator.wikimedia.org/T365975#9895021 (10fnegri) 05Open→03Resolved a:03fnegri