[00:59:36] (03open) 10jjmc89: autovoice stashbot in #wikimedia-hackathon [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/11 [00:59:45] (03update) 10jjmc89: autovoice stashbot in #wikimedia-hackathon [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/11 [01:03:17] (03merge) 10bd808: autovoice stashbot in #wikimedia-hackathon [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/11 (owner: 10jjmc89) [01:19:38] (03open) 10jjmc89: add #wikimedia-cloud* channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/12 (https://phabricator.wikimedia.org/T377744) [01:20:28] (03update) 10jjmc89: add #wikimedia-cloud* channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/12 (https://phabricator.wikimedia.org/T377744) [01:22:34] 06cloud-services-team, 10ircservserv, 13Patch-For-Review: Use ircservserv to manage permissions for #wikimedia-cloud* channels - https://phabricator.wikimedia.org/T377744#10291376 (10JJMC89) Let me know if anything should be changed. [01:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:28:23] (03update) 10jjmc89: add #wikimedia-cloud* channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/12 (https://phabricator.wikimedia.org/T377744) [05:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:30:42] (03approved) 10sstefanova: jobs-api: bump to 0.0.339-20241104094933-bb9cbca1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/571 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [07:30:45] (03merge) 10sstefanova: jobs-api: bump to 0.0.339-20241104094933-bb9cbca1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/571 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [07:32:11] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/75 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:32:15] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/75 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:38:27] (03update) 10sstefanova: calico: bump to 0.0.15-20241104101859-e2c4ee9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/573 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [07:38:28] (03update) 10sstefanova: calico: bump to 0.0.15-20241104101859-e2c4ee9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/573 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [07:39:00] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [07:44:26] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico [07:44:59] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico [07:49:27] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico [07:49:52] (03approved) 10sstefanova: calico: bump to 0.0.15-20241104101859-e2c4ee9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/573 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [07:49:55] (03merge) 10sstefanova: calico: bump to 0.0.15-20241104101859-e2c4ee9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/573 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [07:56:29] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/32 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:56:39] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/32 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:57:28] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/565 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:57:29] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/565 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:04:33] (03approved) 10sstefanova: toolforge_deploy_mr: added restart of deployments if no diff in chart [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/207 (owner: 10dcaro) [08:05:01] (03update) 10sstefanova: builds-builder: bump to 0.0.122-20241104102412-74466167 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/575 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:05:01] (03update) 10sstefanova: builds-builder: bump to 0.0.122-20241104102412-74466167 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/575 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:05:30] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [08:11:01] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [08:11:14] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/32 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:11:24] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/32 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:16:10] 10Cloud-VPS (Quota-requests): Temporary (1-2 weeks) quota increase for disaster recovery exercise - https://phabricator.wikimedia.org/T375977#10291622 (10Slst2020) 05In progress→03Resolved ` Done – thank you! sstefanova@cloudcontrol1005:~$ sudo wmcs-openstack database quota show mwoffliner +-----------+... [08:17:35] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [08:20:10] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/33 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:20:10] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/33 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:20:16] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/33 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:20:41] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/34 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:22:33] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 (https://phabricator.wikimedia.org/T356261) [08:23:22] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [08:24:21] (03approved) 10sstefanova: builds-builder: bump to 0.0.122-20241104102412-74466167 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/575 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:24:23] (03merge) 10sstefanova: builds-builder: bump to 0.0.122-20241104102412-74466167 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/575 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:24:49] (03update) 10sstefanova: wmcs-k8s-metrics: bump to 0.0.21-20241104102245-a6f60a0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/574 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:24:51] (03update) 10sstefanova: wmcs-k8s-metrics: bump to 0.0.21-20241104102245-a6f60a0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/574 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:27:01] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [08:32:20] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics [08:32:45] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [08:38:19] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics [08:39:38] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/46 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:47:07] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/46 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:47:08] (03close) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/46 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:47:44] (03approved) 10sstefanova: wmcs-k8s-metrics: bump to 0.0.21-20241104102245-a6f60a0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/574 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:47:47] (03merge) 10sstefanova: wmcs-k8s-metrics: bump to 0.0.21-20241104102245-a6f60a0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/574 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:52:21] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/31 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:52:40] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/31 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:53:28] (03close) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/31 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:55:54] (03merge) 10dcaro: toolforge_deploy_mr: added restart of deployments if no diff in chart [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/207 [08:59:00] 10Toolforge (Toolforge iteration 16): [toolforge,grafana,infra] Grafana stopped showing the namespaces (and other stats) for toolforge namespaces - https://phabricator.wikimedia.org/T378981#10291671 (10dcaro) This is likely https://github.com/kubernetes/kube-state-metrics/issues/2297 [09:03:12] (03open) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/67 [09:11:12] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/127 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:11:19] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/127 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:16:30] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.32 - https://phabricator.wikimedia.org/T379047 (10dcaro) 03NEW p:05Triage→03High [09:17:05] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.31 - https://phabricator.wikimedia.org/T372697#10291722 (10dcaro) p:05Triage→03High [09:18:33] (03approved) 10dcaro: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/67 (owner: 10sstefanova) [09:19:05] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.32 - https://phabricator.wikimedia.org/T379047#10291756 (10dcaro) This we might want to wait until 1.33 gets released, might be the first we do at the upstream pace :) [09:21:16] (03merge) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/67 [09:22:37] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:24:44] (03approved) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/127 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:24:48] (03merge) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/127 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:25:33] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/66 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:25:33] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/66 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:27:42] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.340-20241105092501-7b7dac5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/577 [09:28:12] (03update) 10sstefanova: jobs-api: bump to 0.0.340-20241105092501-7b7dac5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/577 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:28:27] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:33:25] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/66 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:33:28] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/66 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:34:01] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/65 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:34:07] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:34:16] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:35:56] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/65 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:36:54] (03close) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/65 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:40:22] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:40:48] (03approved) 10sstefanova: jobs-api: bump to 0.0.340-20241105092501-7b7dac5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/577 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:40:51] (03merge) 10sstefanova: jobs-api: bump to 0.0.340-20241105092501-7b7dac5c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/577 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:48:13] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] Develop the webhook mechanism to trigger a deployment - https://phabricator.wikimedia.org/T362066#10291827 (10dcaro) [09:48:58] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with single continuous job deployment only - https://phabricator.wikimedia.org/T362069#10291828 (10dcaro) [09:49:38] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with single continuous job deployment only - https://phabricator.wikimedia.org/T362069#10291830 (10dcaro) I think this... [09:49:41] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/92 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:53:29] 06cloud-services-team, 10Toolforge: [components-api] Add webservice support (to refine) - https://phabricator.wikimedia.org/T362077#10291842 (10Slst2020) [09:53:36] 06cloud-services-team, 10Toolforge: [components-api] Extend the list of build triggers (unrefined) - https://phabricator.wikimedia.org/T362071#10291845 (10Slst2020) [09:56:38] 06cloud-services-team, 10Toolforge: [components-api] Add minimal cli with build-only features - https://phabricator.wikimedia.org/T362082#10291840 (10Slst2020) [09:57:16] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with single continuous job deployment only - https://phabricator.wikimedia.org/T362069#10291837 (10Slst2020) 05In... [09:57:18] 10Toolforge (Toolforge iteration 16): [components-api] Add support for pre-built images (ex. python3.11, to refine) - https://phabricator.wikimedia.org/T362076#10291849 (10dcaro) 05Open→03In progress [10:02:43] (03open) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/94 [10:02:48] (03approved) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/94 [10:02:51] (03merge) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/94 [10:03:11] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/92 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:12:12] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/92 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:12:14] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/92 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:13:39] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/93 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:14:25] (03update) 10dcaro: add token validation [repos/cloud/toolforge/components-api] (add_creation_date_to_token) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/32 (https://phabricator.wikimedia.org/T362066) [10:16:35] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/93 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:16:37] (03close) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/93 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:21:07] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/34 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:21:07] (03approved) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/34 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:21:11] (03merge) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/34 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:23:24] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/565 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:32:18] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#10291962 (10dcaro) We will use custom deploy tokens for deployments, and now horizon has moved to using idp, so that reduces the options. I'll move this back to the queu... [10:32:31] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#10291963 (10dcaro) a:05dcaro→03None [10:33:54] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#10291965 (10dcaro) [10:38:46] 10Toolforge (Toolforge iteration 16): [components-api] Use an asynchronous toolforge client to interact with toolforge - https://phabricator.wikimedia.org/T379053 (10dcaro) 03NEW [10:39:11] 10Toolforge (Toolforge iteration 16): [components-api] Use an asynchronous toolforge client to interact with toolforge - https://phabricator.wikimedia.org/T379053#10291991 (10dcaro) [10:39:17] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10291992 (10dcaro) [10:39:21] 10Toolforge (Toolforge iteration 16): [components-api] Use an asynchronous toolforge client to interact with toolforge - https://phabricator.wikimedia.org/T379053#10291993 (10dcaro) p:05Triage→03High [10:39:51] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#10291994 (10dcaro) 05In progress→03Open [10:44:18] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 (https://phabricator.wikimedia.org/T356261) [10:52:45] (03open) 10sstefanova: ci: upgrade to py311 image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/578 [10:53:06] 10wikitech.wikimedia.org, 06SRE, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10292031 (10jijiki) >>! In T376400#10287901, @MatthewVernon wrote: > @jijiki can you expand on what you mean, please? This task is currently too broad... For the time being the task is delibe... [10:54:31] (03open) 10dcaro: toolforge_deploy_mr: add basic autocompletion [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/208 [10:56:01] (03approved) 10dcaro: ci: upgrade to py311 image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/578 (owner: 10sstefanova) [10:57:20] (03merge) 10sstefanova: ci: upgrade to py311 image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/578 [10:57:23] (03update) 10sstefanova: ci: upgrade to py311 image [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/578 [10:58:21] (03approved) 10sstefanova: toolforge_deploy_mr: add basic autocompletion [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/208 (owner: 10dcaro) [10:58:25] (03update) 10sstefanova: toolforge_deploy_mr: add basic autocompletion [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/208 (owner: 10dcaro) [10:59:04] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/565 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:59:05] (03merge) 10dcaro: toolforge_deploy_mr: add basic autocompletion [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/208 [10:59:23] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/565 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:59:28] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/565 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [11:03:51] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10292062 (10roti_WMDE) |**Wikitech account/LDAP:**| Robert Timm (WMDE) | |**SUL account**| Robert Timm (WMDE) | |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |Y| |**I have vi... [11:17:00] 10wikitech.wikimedia.org, 06SRE, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10292157 (10jijiki) [11:17:21] 10wikitech.wikimedia.org, 06SRE, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10292158 (10jijiki) @MatthewVernon updated description [11:23:44] 06cloud-services-team, 10Cloud-VPS: Frequent radosgw 500 errors with OpenTofu - https://phabricator.wikimedia.org/T360626#10292165 (10Raymond_Ndibe) The priority of this should be high. This basically makes buckets unusable now. I already tried creating and experimenting with two buckets and I can't even push... [11:30:44] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10292196 (10fnegri) > might be related to this upgrade? Potentially yes. Looking at the logs you pasted in T378976, my very vague and unhelpful theo... [11:40:16] 06cloud-services-team, 10Cloud-VPS: Frequent radosgw 500 errors with OpenTofu - https://phabricator.wikimedia.org/T360626#10292238 (10fnegri) @Raymond_Ndibe can you paste one or more example commands that are failing? It's interesting that for OpenTofu it seems to only fail some times but not always. We can... [11:49:29] 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378848#10292279 (10fnegri) a:03komla [12:34:46] 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#10292421 (10Samwalton9-WMF) >>! In T342398#10285184, @taavi wrote: > Next up I think is to find a real project with real traffic... [12:34:48] 06cloud-services-team, 10Cloud-VPS: Frequent radosgw 500 errors with OpenTofu - https://phabricator.wikimedia.org/T360626#10292422 (10dcaro) This might be related to this errors on logstash https://logstash.wikimedia.org/goto/c7fa935688ccd6ccda0e11b420b747d1 ` [None req-ec867736-f8b3-40cd-aeef-fb2ee2daf81c sw... [13:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:43:33] (03CR) 10FNegri: [C:03+1] "I double checked and it should be possible to implement these checks in tofu at "plan" stage:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084042 (owner: 10David Caro) [13:45:13] (03CR) 10FNegri: [C:03+1] vps.create_project: add the checks mentioned in the wiki (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1084042 (owner: 10David Caro) [13:52:40] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate deployment-poolcounter06.deployment-prep.eqiad.wmflabs is about to expire in 22d 23h 58m 30s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [14:00:34] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10292656 (10aborrero) Some good progress today. I was able to get designate-sink to run the new code. The config in `/etc/designate/designate.conf`: ` [service:sink] enabled_notifica... [14:10:48] 06cloud-services-team, 10Cloud-VPS: Remove tofu-infra-test project - https://phabricator.wikimedia.org/T379076 (10rook) 03NEW [14:19:21] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10292732 (10rook) 05Open→03Resolved a:03rook [14:25:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:36:49] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/21 [14:37:53] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1087476 (owner: 10L10n-bot) [14:45:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:50:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:59:56] (03open) 10sstefanova: tests: record new cassettes [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/64 [15:02:48] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] (slavina/record-new-cassettes) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/63 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:03:02] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] (slavina/record-new-cassettes) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/63 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:15:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:16:13] (03approved) 10dcaro: tests: record new cassettes [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/64 (owner: 10sstefanova) [15:18:36] 10Tools: Help needed to deploy a react tool in Toolforge - https://phabricator.wikimedia.org/T374304#10292923 (10dcaro) Hi @Ederporto, were you able to make it work? [15:20:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:45:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:50:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:02:01] 10Tool-refill: refill: review citoid usage - https://phabricator.wikimedia.org/T378686#10293142 (10akosiaris) Peak traffic for refill was on Oct 10 (graph pasted below) so what happened after that isn't particularly important. And refill is apparently already running in another node. {F57683510} > Do these... [16:02:25] 10Cloud-VPS (Quota-requests): Request floating IP for wikiwho project - https://phabricator.wikimedia.org/T376637#10293172 (10aborrero) Yes, I think I agree! hey @MusikAnimal would you be interested in testing this new feature? [16:15:28] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-cli] Create cli with subcommand - https://phabricator.wikimedia.org/T379091 (10dcaro) 03NEW [16:15:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:17:29] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-api] Add functional tests for the components api - https://phabricator.wikimedia.org/T379092 (10dcaro) 03NEW [16:18:02] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-cli] Create cli with subcommand - https://phabricator.wikimedia.org/T379091#10293247 (10dcaro) [16:18:07] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-api] Add functional tests for the components api - https://phabricator.wikimedia.org/T379092#10293248 (10dcaro) [16:18:11] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10293249 (10dcaro) [16:19:01] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10293251 (10Ladsgroup) Hi, can you try the 2fa value for your SUL account? [16:19:55] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-api] Add endpoint to delete a deployment - https://phabricator.wikimedia.org/T379093 (10dcaro) 03NEW [16:21:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:22:46] (03merge) 10sstefanova: tests: record new cassettes [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/64 [16:22:49] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/63 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:24:54] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-api] Add functional tests for the components api - https://phabricator.wikimedia.org/T379092#10293288 (10dcaro) p:05Triage→03High [16:25:12] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.170-20241105162258-f4580ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/579 [16:25:16] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.170-20241105162258-f4580ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/579 [16:25:40] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-cli] Create cli with subcommand - https://phabricator.wikimedia.org/T379091#10293290 (10Slst2020) a:03Slst2020 [16:27:27] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-api] Add endpoint to delete a deployment - https://phabricator.wikimedia.org/T379093#10293295 (10dcaro) p:05Triage→03High [16:27:31] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-cli] Create cli with subcommand - https://phabricator.wikimedia.org/T379091#10293297 (10dcaro) p:05Triage→03High [16:27:44] 10Toolforge (Toolforge iteration 16): [components-api] Use an asynchronous toolforge client to interact with toolforge - https://phabricator.wikimedia.org/T379053#10293298 (10dcaro) a:03dcaro [16:29:44] 10Toolforge (Toolforge iteration 16): [components-api] Use an asynchronous toolforge client to interact with toolforge - https://phabricator.wikimedia.org/T379053#10293300 (10dcaro) 05Open→03In progress [16:31:28] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10293313 (10aborrero) seems to be working as expected: `lang=shell-session aborrero@bastion-codfw1dev-04:~$ host fullstackd-20241105162427.admin-monitoring.codfw1dev.wikimedia.cloud f... [16:46:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:50:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:56:11] 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#10293466 (10taavi) >>! In T342398#10292421, @Samwalton9-WMF wrote: >>>! In T342398#10285184, @taavi wrote: >> Next up I think is... [17:06:03] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [17:07:53] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [components-api] Add endpoint to delete a deployment - https://phabricator.wikimedia.org/T379093#10293554 (10Slst2020) a:03Slst2020 [17:12:24] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [17:12:26] 06cloud-services-team, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q2): Karma UI shows duplicate alerts - https://phabricator.wikimedia.org/T353457#10293598 (10lmata) [17:12:52] (03update) 10sstefanova: maintain-kubeusers: bump to 0.0.170-20241105162258-f4580ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/579 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:13:47] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [17:15:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:16:03] 06cloud-services-team, 10Cloud-VPS, 10observability, 10Observability-Logging, and 3 others: ossl rsyslog errors post-migration - https://phabricator.wikimedia.org/T351710#10293600 (10lmata) [17:20:22] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [17:20:32] (03approved) 10sstefanova: maintain-kubeusers: bump to 0.0.170-20241105162258-f4580ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/579 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:20:35] (03merge) 10sstefanova: maintain-kubeusers: bump to 0.0.170-20241105162258-f4580ef8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/579 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:20:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:22:40] 10Cloud-VPS (Quota-requests): Request floating IP for wikiwho project - https://phabricator.wikimedia.org/T376637#10293654 (10MusikAnimal) >>! In T376637#10293172, @aborrero wrote: > Yes, I think I agree! > > hey @MusikAnimal would you be interested in testing this new feature? I have asked the owners of wikiw... [17:30:07] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/63 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:30:10] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/63 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:30:19] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/63 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:32:47] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.171-20241105173021-bf5186a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/580 [17:45:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:46:44] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: Improve WMCS NodeDown alerts - https://phabricator.wikimedia.org/T375479#10293769 (10fnegri) [17:48:22] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: 2024-09-21 NodeDown cloudvirt1063 - https://phabricator.wikimedia.org/T375223#10293772 (10fnegri) 05Stalled→03In progress [17:49:57] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: Improve WMCS NodeDown alerts - https://phabricator.wikimedia.org/T375479#10293763 (10fnegri) I believe the current set of patches should at least fix the two issues in the task description: > a cloudvirt going down is triggering 3 a... [17:51:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:52:04] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: 2024-09-21 NodeDown cloudvirt1063 - https://phabricator.wikimedia.org/T375223#10293778 (10fnegri) 05In progress→03Stalled We are waiting for Dell to replace the mainboard, see the subtask {T375372} [18:07:50] 06cloud-services-team, 10Cloud-VPS: Delete project tf-infra-test - https://phabricator.wikimedia.org/T376890#10293874 (10fnegri) →14Duplicate dup:03T379076 [18:09:09] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10293876 (10fnegri) [18:11:17] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10293888 (10fnegri) 05Resolved→03Open Unfortunately OpenStack is terrible and deleting a project does not seem to delete the VMs that were in that project. I noticed this because the `/usr/local/bin... [18:14:13] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10293892 (10fnegri) Now I'm not sure how to find the names/ids of the VMs that were part of that project, because if I try `wmcs-openstack server list --project tf-infra-test` it fails with `No project... [18:15:09] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10293895 (10rook) Things that are good to know. I'll see what I can find [18:16:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:21:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:28:44] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/21 (owner: 10l10n-bot) [18:28:46] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/21 (owner: 10l10n-bot) [18:38:38] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10293941 (10rook) I believe 40560d4a-6b06-49be-bfcd-2565666ef95d is our system: ` openstack server show 40560d4a-6b06-49be-bfcd-2565666ef95d +-------------------------------------+---------------------... [18:39:32] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10293942 (10rook) Do we feel that running `openstack server delete 40560d4a-6b06-49be-bfcd-2565666ef95d` would be safe? [18:46:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:50:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:52:49] !log fnegri@cloudcumin1001 catalyst START - Cookbook wmcs.openstack.quota_increase (T378848) [18:52:52] T378848: Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378848 [18:52:57] !log fnegri@cloudcumin1001 catalyst END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T378848) [18:57:04] (03update) 10bd808: add #wikimedia-cloud* channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/12 (https://phabricator.wikimedia.org/T377744) (owner: 10jjmc89) [18:57:29] FIRING: InstanceDown: Project cloudinfra instance cloud-cumin-03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:59:32] !log fnegri@cloudcumin1001 catalyst START - Cookbook wmcs.openstack.quota_increase (T378231) [18:59:35] T378231: Increase catalyst storage to 550GB - https://phabricator.wikimedia.org/T378231 [18:59:40] !log fnegri@cloudcumin1001 catalyst END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T378231) [19:01:06] 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378848#10294062 (10komla) This has been done. [19:01:18] 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378848#10294063 (10komla) 05Open→03Resolved [19:01:39] 10Cloud-VPS (Quota-requests): Increase catalyst storage to 550GB - https://phabricator.wikimedia.org/T378231#10294067 (10komla) This has been done [19:01:48] 10Cloud-VPS (Quota-requests): Increase catalyst storage to 550GB - https://phabricator.wikimedia.org/T378231#10294068 (10komla) 05Open→03Resolved [19:02:29] RESOLVED: InstanceDown: Project cloudinfra instance cloud-cumin-03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:06:03] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10294075 (10fnegri) > Do we feel that running openstack server delete 40560d4a-6b06-49be-bfcd-2565666ef95d would be safe? I think so... but the openstack API might have different opinions. I think it's... [19:09:14] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10294078 (10rook) Looks like it took ` openstack server show 40560d4a-6b06-49be-bfcd-2565666ef95d No Server found for 40560d4a-6b06-49be-bfcd-2565666ef95d ` [19:12:43] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10294079 (10fnegri) Can you try re-running `/usr/local/bin/wmcs-dnsleaks` from a cloudcontrol? [19:15:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [19:17:30] 06cloud-services-team, 10Cloud-VPS: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076#10294084 (10rook) I don't see that file on either cloudcontrol1005.eqiad.wmnet or cloudcontrol1007.eqiad.wmnet [19:40:55] vivian-rook opened https://github.com/toolforge/paws/pull/461 [21:17:30] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10294409 (10Eevans) |**Wikitech account/LDAP:**| eevans| |**SUL account**| EEvans (WMF)| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |Y| |**I have visited [[ https://wikitec... [21:18:13] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:22:37] FIRING: CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:37:57] 10VPS-project-Wikistats: add ae.wikimedia.org to wikistats - https://phabricator.wikimedia.org/T369858#10294493 (10Dzahn) ` MariaDB [wikistats]> insert into wmspecials (lang,prefix,statsurl) values ("Arabic","ae","https://ae.wikimedia.org/w/api.php?action=query&meta=siteinfo&siprop=statistics"); ` [21:39:54] 10VPS-project-Wikistats: add ae.wikimedia.org to wikistats - https://phabricator.wikimedia.org/T369858#10294505 (10Dzahn) Don't forget to set method to 8 for everything to use the "new" way to fetch stats via the normal API, or it won't work. ` MariaDB [wikistats]> update wmspecials set method=8; ` ` dzahn@w... [21:42:14] 10VPS-project-Wikistats: add ae.wikimedia.org to wikistats - https://phabricator.wikimedia.org/T369858#10294509 (10Dzahn) 05Open→03Resolved wmspecial wikis need a "description" field. ` MariaDB [wikistats]> update wmspecials set description="Wikimedians of United Arab Emirates User Group" where prefix=... [21:42:59] 06cloud-services-team, 10Data-Services, 10Wiki-Setup (Create): Create a Wikimedians of United Arab Emirates User Group Wiki - https://phabricator.wikimedia.org/T362529#10294512 (10Dzahn) [22:21:20] (03PS2) 10AntiCompositeNumber: Handle temporary accounts as anons [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) [22:21:41] (03CR) 10AntiCompositeNumber: Handle temporary accounts as anons (031 comment) [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1084298 (https://phabricator.wikimedia.org/T378530) (owner: 10AntiCompositeNumber) [23:43:17] 06cloud-services-team, 10Cloud-VPS: OpenStack project ids rather than names are being used in hiera settings archiving - https://phabricator.wikimedia.org/T379128 (10bd808) 03NEW [23:43:47] 06cloud-services-team, 10Cloud-VPS: OpenStack project ids rather than names are being used in hiera settings archiving - https://phabricator.wikimedia.org/T379128#10294811 (10bd808) [23:43:53] 06cloud-services-team, 10Cloud-VPS, 07Epic: Wind down use of project ID and project name equivalency in OpenStack - https://phabricator.wikimedia.org/T274268#10294812 (10bd808)