[00:00:16] (03update) 10bd808: Draft: Initial commit for frontend; work in progress [toolforge-repos/toolviews] - 10https://gitlab.wikimedia.org/toolforge-repos/toolviews/-/merge_requests/8 (owner: 10musikanimal) [00:00:24] (03update) 10bd808: Draft: Initial commit for frontend; work in progress [toolforge-repos/toolviews] - 10https://gitlab.wikimedia.org/toolforge-repos/toolviews/-/merge_requests/8 (owner: 10musikanimal) [02:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:59:49] (03PS1) 10Brian Wolff: Ensure file name uses 7 character rev even if ambigious [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/1034212 (https://phabricator.wikimedia.org/T365416) [05:31:21] (03CR) 10Brian Wolff: "I have an alternative patch at https://gerrit.wikimedia.org/r/c/labs/tools/extdist/+/1034212" [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/937474 (https://phabricator.wikimedia.org/T340882) (owner: 10Reedy) [05:51:27] dependabot[bot] opened https://github.com/toolforge/paws/pull/413 [06:39:56] 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424 (10Marostegui) 03NEW [06:42:10] 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9815141 (10Marostegui) According to the wiki replicas responsibilities documents, OS upgrades are not performed by #data-persistence but we would be happy to help if something needs our attention. [06:46:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:49:25] 10Toolforge: [components-api] add one-off, scheduled and continuous jobs support to the yaml + api - https://phabricator.wikimedia.org/T362075#9815151 (10Slst2020) >>! In T362075#9806795, @Raymond_Ndibe wrote: > I have a thing against `reuse-from`. It is not immediately clear what it means by just looking at it.... [06:50:58] 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9815163 (10Marostegui) @BTullis clouddb1021 belongs to your team, so could you take care of that one? [07:46:27] (03update) 10aborrero: maintain_kubeusers: add support for kyverno policies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 (https://phabricator.wikimedia.org/T279110) [07:47:05] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [07:53:10] 10Data-Services, 07User-notice: Remove deprecated abuse filter fields from Wiki Replicas - https://phabricator.wikimedia.org/T361996#9815373 (10matej_suchanek) 05Open→03Resolved a:03matej_suchanek [07:58:37] (03open) 10jelto: run tests on wmcs runners [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/45 (https://phabricator.wikimedia.org/T362401) [08:18:26] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [08:23:56] FIRING: [4x] NeutronAgentDown: Neutron neutron-linuxbridge-agent on cloudnet1005 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [08:41:06] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#9815592 (10Slst2020) 05Open→03Stalled [08:45:49] 10Toolforge (Toolforge iteration 09), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#9815607 (10Slst2020) Robot accounts still don't have update permissions on project quotas. After bringing it up in the CNCF harbor chan... [08:54:52] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [09:01:55] (03open) 10sstefanova: api-gateway: remove wait condition [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/131 [09:15:23] (03approved) 10sstefanova: [lima-kilo] enable toolforge-weld installation [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/130 (owner: 10raymond-ndibe) [09:23:49] (03approved) 10aborrero: api-gateway: remove wait condition [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/131 (owner: 10sstefanova) [09:30:55] (03merge) 10sstefanova: api-gateway: remove wait condition [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/131 [09:33:04] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [09:37:29] 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9815773 (10BTullis) >>! In T365424#9815162, @Marostegui wrote: > @BTullis clouddb1021 belongs to your team, so could you take care of that one? Yes, will do. [09:46:32] 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Upgrade clouddb1021 to bookworm - https://phabricator.wikimedia.org/T365450 (10BTullis) 03NEW [09:46:55] 10Toolforge: [builds-api] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363808#9815833 (10Slst2020) a:03Slst2020 [09:48:17] 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Upgrade clouddb1021 to bookworm - https://phabricator.wikimedia.org/T365450#9815838 (10Marostegui) By default `/srv` is preserved `modules/profile/data/profile/installserver/preseed.yaml:` ` 'clouddb1*': - reuse-parts.cfg - partman/custom/... [09:48:18] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [09:49:13] 10Toolforge (Toolforge iteration 09): [builds-api] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363808#9815837 (10Slst2020) [09:51:52] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9815857 (10taavi) [09:55:54] (03CR) 10Arturo Borrero Gonzalez: [C:03+1] "LGTM." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1034089 (https://phabricator.wikimedia.org/T364459) (owner: 10Majavah) [09:56:11] (03CR) 10Majavah: [C:03+2] openstack: cloudnet: Add one-off cookbook for OVS migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1034089 (https://phabricator.wikimedia.org/T364459) (owner: 10Majavah) [09:59:17] (03Merged) 10jenkins-bot: openstack: cloudnet: Add one-off cookbook for OVS migration [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1034089 (https://phabricator.wikimedia.org/T364459) (owner: 10Majavah) [10:00:37] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 cloudnets to Neutron OVS agent - https://phabricator.wikimedia.org/T364459#9815904 (10taavi) [10:31:39] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [10:37:20] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9816003 (10fnegri) a:03fnegri I can do the reimages for the WMCS hosts. A few questions: > Stop mariadb (instance by instance, DO NOT DO systemctl stop mariadb@s*) W... [10:39:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-idp-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:46:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:03:31] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T365459 (10Keith_D) 03NEW [11:04:53] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete wmflabs dummy certs [labs/private] - 10https://gerrit.wikimedia.org/r/1032713 (owner: 10Muehlenhoff) [11:06:49] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [11:11:10] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 cloudnets to Neutron OVS agent - https://phabricator.wikimedia.org/T364459#9816126 (10taavi) [11:12:23] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs [11:42:28] RESOLVED: [2x] MetricsinfraAlertmanagerDown: Metricsinfra alertmanager is unreachable #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MetricsinfraAlertmanagerDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DMetricsinfraAlertmanagerDown [11:47:23] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9816227 (10Marostegui) >>! In T365424#9816003, @fnegri wrote: > I can do the reimages for the WMCS hosts. > > A few questions: > >> Stop mariadb (instance by instance,... [11:47:34] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [11:49:17] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [12:04:37] 10PAWS: upgrade ingress-nginx - https://phabricator.wikimedia.org/T365386#9816298 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/412 [12:04:38] 10PAWS: upgrade ingress-nginx - https://phabricator.wikimedia.org/T365386#9816299 (10rook) 05Open→03Resolved [12:04:43] vivian-rook closed https://github.com/toolforge/paws/pull/412 [12:10:16] 10Quarry: [bug] My queries are not being executed - https://phabricator.wikimedia.org/T365468 (10Magnus) 03NEW [12:16:36] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 cloudnets to Neutron OVS agent - https://phabricator.wikimedia.org/T364459#9816334 (10taavi) [12:33:41] (03open) 10sstefanova: dev: oapi-codegen updates [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/91 [12:41:59] (03update) 10sstefanova: dev: oapi-codegen updates [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/91 [12:43:19] (03update) 10sstefanova: dev: oapi-codegen updates [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/91 [12:53:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [12:58:13] (03open) 10sstefanova: dev: prevent oapi-codegen version mismatch [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/92 [13:24:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-idp-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:42:38] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [13:42:43] 10Quarry: Queued Quarry queries can't be stopped - https://phabricator.wikimedia.org/T365477 (10MBH) 03NEW [13:45:02] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 cloudnets to Neutron OVS agent - https://phabricator.wikimedia.org/T364459#9816551 (10taavi) 05In progress→03Resolved [13:57:17] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [14:10:35] vivian-rook closed https://github.com/toolforge/paws/pull/413 [14:11:55] (03open) 10sstefanova: Draft: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/93 [14:38:21] (03approved) 10raymond-ndibe: dev: prevent oapi-codegen version mismatch [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/92 (owner: 10sstefanova) [14:38:24] (03update) 10raymond-ndibe: dev: prevent oapi-codegen version mismatch [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/92 (owner: 10sstefanova) [14:41:57] (03merge) 10sstefanova: dev: prevent oapi-codegen version mismatch [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/92 [14:42:56] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-prep kafka hosts with Bullseye or Bookworm - https://phabricator.wikimedia.org/T361382#9816916 (10elukey) All nodes should be on Bullseye, but I have manually upgraded them. @Andrew do we need to... [14:46:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:47:38] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.144-20240521144209-4947025a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/283 [14:49:56] 10Toolforge (Toolforge iteration 09): [builds-api] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363808#9816969 (10Slst2020) 05Open→03In progress [15:11:24] (03update) 10sstefanova: Draft: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/93 [15:28:40] 10Quarry: quarry down - https://phabricator.wikimedia.org/T365496 (10rook) 03NEW [15:29:28] 10Quarry: quarry down - https://phabricator.wikimedia.org/T365496#9817337 (10rook) Looks like none of the web pods are running. Logs give ` [2024-05-21 15:28:04 +0000] [11] [ERROR] Error handling request / Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py"... [15:31:00] 10Quarry: quarry down - https://phabricator.wikimedia.org/T365496#9817358 (10rook) 05Open→03Resolved [15:31:20] 10Quarry: quarry down - https://phabricator.wikimedia.org/T365496#9817352 (10rook) @fnegri fixed it [15:51:51] 10Quarry: quarry down - https://phabricator.wikimedia.org/T365496#9817466 (10rook) 05Resolved→03Open [15:52:07] 10Quarry: quarry down - https://phabricator.wikimedia.org/T365496#9817467 (10rook) Still appears to be down. Web page is loading, but queries are giving: `Access denied for user 'quarry'@'172.16.2.72' (using password: NO)` [15:53:38] 10Quarry: quarry down - https://phabricator.wikimedia.org/T365496#9817482 (10rook) Restarting the services seems to have things connecting again. `kubectl rollout restart deployment.apps/redis deployment.apps/web deployment.apps/worker` [15:53:44] 10Quarry: quarry down - https://phabricator.wikimedia.org/T365496#9817484 (10rook) 05Open→03Resolved [15:54:20] (03update) 10sstefanova: Draft: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/93 [15:59:35] vivian-rook opened https://github.com/toolforge/quarry/pull/41 [15:59:37] 10Quarry: Error 500 when clicking "stop query" - https://phabricator.wikimedia.org/T362213#9817568 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/41 [16:06:24] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry, 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9817624 (10fnegri) > I'm not sure how to ask Quarry to connect to ToolsDB instead of the wikirepl... [16:09:44] PROBLEM - Check nf_conntrack usage in neutron netns on cloudnet2006-dev is CRITICAL: CRITICAL: no netns defined? https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [16:17:00] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry, 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9817680 (10SD0001) Yes, that issue is T365374. Should be unrelated. [16:19:01] (03PS1) 10BryanDavis: wikitech: Add dummy GitLab API token [labs/private] - 10https://gerrit.wikimedia.org/r/1034533 (https://phabricator.wikimedia.org/T316418) [16:21:28] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry, 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9817723 (10fnegri) > Yes, that issue is T365374. Should be unrelated. T365374 started to happen y... [16:23:17] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [16:25:16] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9817757 (10fnegri) I //always// get this error if I enter a ToolsDB database as the db name (e.g. `s55771__wsstats_p`). [16:25:17] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9817762 (10fnegri) [16:25:22] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry, 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9817761 (10fnegri) [16:27:52] 10Quarry: Error 500 when clicking "stop query" - https://phabricator.wikimedia.org/T362213#9817770 (10SD0001) [16:28:41] 10Quarry: Queued Quarry queries can't be stopped - https://phabricator.wikimedia.org/T365477#9817768 (10SD0001) →14Duplicate dup:03T362213 [16:28:54] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9817777 (10fnegri) [16:30:26] 10Cloud-VPS: Better support for Postgres on Trove - https://phabricator.wikimedia.org/T337396#9817778 (10fnegri) a:05fnegri→03None [16:30:28] 10Cloud-VPS, 10Quarry: [bug] Lot of queries stuck in queued state for hours and days - https://phabricator.wikimedia.org/T365136#9817794 (10SD0001) [16:30:32] 10Quarry: [bug] My queries are not being executed - https://phabricator.wikimedia.org/T365468#9817790 (10SD0001) →14Duplicate dup:03T365136 [16:31:16] 10Cloud-VPS, 10Quarry: [bug] Lot of queries stuck in queued state for hours and days - https://phabricator.wikimedia.org/T365136#9817792 (10SD0001) [16:39:45] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9817824 (10Liz) If you want to see an example of this, try [[ https://quarry.wmcloud.org/history/82726 | Category redirects ]] and look at the history. I could never get this one to run. [16:54:35] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T365459#9817879 (10Curb_Safe_Charmer) @Keith_D it looks fine to me - I know the WMF were doing some maintenance earlier which might have caused a blip. Please confirm it is now working so we can close. [16:59:20] vivian-rook closed https://github.com/toolforge/quarry/pull/41 [16:59:22] 10Quarry: Error 500 when clicking "stop query" - https://phabricator.wikimedia.org/T362213#9817905 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/41 [17:01:00] 10Quarry: Error 500 when clicking "stop query" - https://phabricator.wikimedia.org/T362213#9817908 (10rook) 05Open→03Resolved [17:28:05] 10Tool-bash: Migrate Bash git hosting from github to gitlab - https://phabricator.wikimedia.org/T342919#9818073 (10bd808) 05Stalled→03Open [17:47:42] (03update) 10bd808: Convert CI to gitlab and update to PHP 8.2 [toolforge-repos/bash] - 10https://gitlab.wikimedia.org/toolforge-repos/bash/-/merge_requests/1 (https://phabricator.wikimedia.org/T342919) [17:55:47] (03update) 10bd808: Convert CI to gitlab and update to PHP 8.2 [toolforge-repos/bash] - 10https://gitlab.wikimedia.org/toolforge-repos/bash/-/merge_requests/1 (https://phabricator.wikimedia.org/T342919) [18:02:25] (03update) 10bd808: Convert CI to gitlab and update to PHP 8.2 [toolforge-repos/bash] - 10https://gitlab.wikimedia.org/toolforge-repos/bash/-/merge_requests/1 (https://phabricator.wikimedia.org/T342919) [18:05:50] RECOVERY - Check nf_conntrack usage in neutron netns on cloudnet2006-dev is OK: OK: everything is apparently fine https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [18:06:27] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T365459#9818351 (10Keith_D) Looks like working now. Thanks. [18:07:52] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T365459#9818353 (10Pppery) 05Open→03Resolved Sigh. [18:09:00] 10Cloud-VPS, 13Patch-Needs-Improvement, 07Puppet: role::puppetmaster::standalone clones Git repositories as gitpuppet, git-sync-upstream overwrites them as root - https://phabricator.wikimedia.org/T152059#9818362 (10Pppery) [18:46:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:23:25] 10Cloud-Services, 06DC-Ops, 10ops-eqiad, 06SRE: Degraded RAID on cloudcephosd1031 - https://phabricator.wikimedia.org/T364060#9818685 (10Dzahn) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it w... [19:25:12] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 05Cloud-Services-Origin-Alert: Degraded RAID on cloudcephosd1031 - https://phabricator.wikimedia.org/T364060#9818688 (10RhinosF1) [19:54:32] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, and 2 others: Degraded RAID on cloudcephosd1031 - https://phabricator.wikimedia.org/T364060#9818758 (10taavi) [22:01:55] 10Toolforge (Toolforge iteration 09): [maintain-kubeusers] Increment default services quota - https://phabricator.wikimedia.org/T362520#9819219 (10Raymond_Ndibe) a:03Raymond_Ndibe [22:45:32] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T365536 (10derenrich) 03NEW [22:46:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:58:34] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T365536#9819329 (10derenrich) For context on the 5GB number. A single build after a clean leaves my quota at 928MiB used. [23:03:10] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T365536#9819340 (10bd808) {F54094708, size=full} ` $ toolforge build quota Registry =================== Storage ----------- Available 95.58Mi Capacity 91% Limit 1.00Gi Used... [23:06:11] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T365536#9819341 (10bd808) It might also be vaguely interesting to look into any changes to https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/apt-buildpack that might reduce t... [23:09:12] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9819342 (10Liz) Thanks to whomever fixed this problem. [23:22:47] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T365536#9819355 (10derenrich) sounds good [23:27:12] 06cloud-services-team, 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T365536#9819364 (10bd808) p:05Triage→03Medium +1 for trying this with a 2GB quota