[01:01:46] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:21:46] FIRING: [3x] CloudVPSDesignateLeaks: Detected 70 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:29:22] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211#9811627 (10Bugreporter) I am not planning to reopen the task for now, but @bd808 may have another opinion. Anyway, see my comment at T102576 - In fact, if the activity field of PhabBanBot in Phabricator displa... [02:41:46] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [03:23:55] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:21:46] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:21:46] FIRING: [3x] CloudVPSDesignateLeaks: Detected 72 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:45:23] 10Data-Services, 06DBA: Prepare and check storage layer for dtpwiki - https://phabricator.wikimedia.org/T365229#9811692 (10Marostegui) 05Open→03Resolved p:05Triage→03Medium a:03Marostegui Let know us know when the wiki is created so we can sanitize it [05:46:32] 10Data-Services, 06DBA: Prepare and check storage layer for dtpwiki - https://phabricator.wikimedia.org/T365229#9811698 (10Marostegui) 05Resolved→03Open a:05Marostegui→03None [06:46:46] FIRING: [4x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:16:51] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [08:20:08] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [08:21:46] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [08:32:04] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [08:51:46] FIRING: [5x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:58:31] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [09:07:11] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [09:08:24] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [09:21:46] FIRING: [3x] CloudVPSDesignateLeaks: Detected 70 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:51:46] FIRING: [3x] CloudVPSDesignateLeaks: Detected 71 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:53:55] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 71 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:20:45] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [10:20:55] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [10:31:03] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry, 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9812034 (10github-toolforge-bot) dhinus opened https://github.com/toolforge/quarry/pull/40 [10:31:05] dhinus opened https://github.com/toolforge/quarry/pull/40 [10:41:38] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [10:41:55] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [10:45:55] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [10:47:40] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [11:00:00] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [11:00:22] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [11:33:27] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [11:44:11] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [11:44:28] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [11:52:50] vivian-rook closed https://github.com/toolforge/quarry/pull/40 [11:52:52] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry, 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9812218 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/40 [12:12:55] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [12:13:11] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [12:18:55] FIRING: [5x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:21:09] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/6 [12:22:10] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1034073 (owner: 10L10n-bot) [12:26:08] 10Quarry: quarry puppet not running - https://phabricator.wikimedia.org/T365357 (10rook) 03NEW [12:40:26] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [12:46:51] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [12:48:26] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1034073 (owner: 10L10n-bot) [13:02:12] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [13:02:23] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [13:06:47] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [13:21:46] FIRING: [4x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:30:30] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/6 (owner: 10l10n-bot) [13:30:37] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/6 (owner: 10l10n-bot) [13:51:30] 10Toolforge: Query appears to run for a longer time when invoked via toolforge jobs framework - https://phabricator.wikimedia.org/T363286#9812551 (10taavi) No, not really.. https://sql-optimizer.toolforge.org gives a good idea why it's so slow (it's doing a filesort on the `page` table as well as going through 3... [14:13:55] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:15:46] !log taavi@runko admin START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs [14:15:46] !log taavi@runko admin END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99) [14:15:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:15:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:16:17] !log taavi@runko admin START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs [14:16:17] !log taavi@runko admin END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99) [14:16:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:16:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:17:40] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [14:17:47] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [14:17:48] !log taavi@runko admin START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs [14:17:48] !log taavi@runko admin END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99) [14:17:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:17:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:18:22] ^ this is all testing on codfw1dev [14:18:25] !log taavi@runko admin START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs [14:18:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:18:45] !log taavi@runko admin END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99) [14:18:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:19:57] !log taavi@runko admin START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs [14:19:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:20:06] !log taavi@runko admin END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0) [14:20:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:21:48] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [14:23:19] !log taavi@runko admin START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs [14:23:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:23:34] !log taavi@runko admin END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0) [14:23:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:24:01] 10Quarry: quarry puppet not running - https://phabricator.wikimedia.org/T365357#9812602 (10rook) 05Open→03Resolved [14:26:41] (03PS1) 10Majavah: openstack: cloudnet: Add one-off cookbook for OVS migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1034089 (https://phabricator.wikimedia.org/T364459) [14:29:16] (03CR) 10CI reject: [V:04-1] openstack: cloudnet: Add one-off cookbook for OVS migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1034089 (https://phabricator.wikimedia.org/T364459) (owner: 10Majavah) [14:31:39] 10PAWS: update minikube instructions - https://phabricator.wikimedia.org/T365364 (10rook) 03NEW [14:32:11] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [14:34:17] (03PS2) 10Majavah: openstack: cloudnet: Add one-off cookbook for OVS migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1034089 (https://phabricator.wikimedia.org/T364459) [14:46:08] !log taavi@runko admin START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs [14:46:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:46:20] !log taavi@runko admin END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0) [14:46:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:56:37] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [15:01:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-10 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:07:59] (03update) 10raymond-ndibe: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 (https://phabricator.wikimedia.org/T348758) [15:14:46] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [15:17:09] (03update) 10raymond-ndibe: [jobs-cli] support services in jobs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/18 (https://phabricator.wikimedia.org/T348758) [15:22:54] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [15:25:58] (03update) 10raymond-ndibe: [envvars-cli] add messages to all responses [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/38 (https://phabricator.wikimedia.org/T356974) [15:31:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-10 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:52:36] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9812917 (10elukey) I have asked something on IRC's `wikimedia-cloud` about a way to speed up the process. In the puppet repo w... [15:52:46] (03update) 10raymond-ndibe: [jobs-api] support services in jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/71 (https://phabricator.wikimedia.org/T348758) [16:06:45] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [16:15:50] (03PS1) 10JHathaway: add mpic dummy secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1034115 [16:16:45] (03CR) 10JHathaway: [C:03+2] add mpic dummy secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1034115 (owner: 10JHathaway) [16:16:49] (03CR) 10JHathaway: [V:03+2 C:03+2] add mpic dummy secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1034115 (owner: 10JHathaway) [16:19:12] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-prep kafka hosts with Bullseye or Bookworm - https://phabricator.wikimedia.org/T361382#9813048 (10elukey) I've done jumbo 5 and 8 with in-place upgrade (so the VM's metadata will need to be fixed). [16:31:16] 10Quarry: [bug] Quarry bug - https://phabricator.wikimedia.org/T365374 (10Liz) 03NEW [16:33:19] 10Quarry: [bug] Quarry bug - https://phabricator.wikimedia.org/T365374#9813121 (10Liz) [16:36:12] 10Quarry: [bug] Quarry bug - https://phabricator.wikimedia.org/T365374#9813142 (10Liz) Sorry, but whatever was the problem is fixed, it's working now. Feel free to close this. Thanks. [16:36:46] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:37:28] 10Quarry: [bug] Quarry bug - https://phabricator.wikimedia.org/T365374#9813145 (10rook) 05Open→03Resolved [17:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:27:59] vivian-rook opened https://github.com/toolforge/paws/pull/411 [17:29:15] vivian-rook closed https://github.com/toolforge/paws/pull/411 [18:09:21] (03merge) 10bd808: Add extloc link [toolforge-repos/versions] - 10https://gitlab.wikimedia.org/toolforge-repos/versions/-/merge_requests/4 (https://phabricator.wikimedia.org/T296050) (owner: 10brennen) [18:36:46] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:41:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:39:57] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9814008 (10SD0001) 05Resolved→03Open Still occurring intermittently. Error message suggests a failure in accessing the Trove db. @rook did we make any changes to the setup lately? [20:00:41] 10Quarry: Error 500 when clicking "stop query" - https://phabricator.wikimedia.org/T362213#9814081 (10SD0001) Turns out the issue is quite simpler - only running queries can be stopped. A query in queued state has no db process associated with it, so there's nothing to kill. I don't think the celery based archit... [20:04:04] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9814111 (10Liz) I spoke too soon. It's happening again. [20:36:10] 10Cloud-VPS, 10Quarry: [bug] Lot of queries stuck in queued state for hours and days (with stop actions leading to HTTP 500) - https://phabricator.wikimedia.org/T365136#9814203 (10SD0001) Per the worker logs, we are getting a lot of connection resets while accessing the trove db: ` sqlalchemy.exc.OperationalE... [20:43:00] 10Cloud-VPS (Debian Buster Deprecation), 06The-Wikipedia-Library, 10Moderator-Tools-Team (Kanban): Replace deprecated Buster VMs in Cloud VPS - https://phabricator.wikimedia.org/T364399#9814227 (10Kgraessle) a:03Kgraessle [20:55:58] 10PAWS: upgrade ingress-nginx - https://phabricator.wikimedia.org/T365386#9814288 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/412 [20:56:07] vivian-rook opened https://github.com/toolforge/paws/pull/412 [21:41:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:36:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-app10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:37:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:37:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tf-bastion on project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:38:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance etcd-discovery-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:39:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:40:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:58:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:59:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:00:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:02:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:04:28] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:08:28] RESOLVED: [4x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:16:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance cvn-nfs-1 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources