[00:23:42] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [00:25:07] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [00:52:01] (03update) 10raymond-ndibe: Draft: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [00:52:43] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [00:53:53] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [02:20:42] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:30:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:32:20] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9952660 (10Slst2020) [05:34:30] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9952661 (10Slst2020) a:03Slst2020 [05:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:07:49] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9952669 (10Slst2020) [06:23:35] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9952672 (10Slst2020) [06:33:02] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9952675 (10Slst2020) The k8s client libraries used in the various admission controllers range from 0.21.x to 0.27.x. My gu... [06:43:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9952677 (10Slst2020) From the reloader docs: ` Reloader is compatible with Kubernetes >= 1.19 ` https://github.com/staka... [06:49:16] 06cloud-services-team, 10wikitech.wikimedia.org: Move Wikitech onto the production MW cluster - https://phabricator.wikimedia.org/T237773#9952697 (10dcausse) Should T192361 be re-opened and added as a subtask here? [07:07:26] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9952729 (10Slst2020) [07:11:02] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25 - https://phabricator.wikimedia.org/T329671#9952751 (10Slst2020) >>! In T329671#9950216, @aborrero wrote: > hey @Slst2020 maybe we can translate that cert-manager and... [07:19:46] (03merge) 10dcaro: create_mrs: skip archived repos [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/42 (https://phabricator.wikimedia.org/T369188) [07:19:47] (03update) 10dcaro: create_mrs: skip archived repos [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/42 (https://phabricator.wikimedia.org/T369188) [07:21:04] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/78 [07:21:08] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/99 [07:21:18] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/50 [07:21:28] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/27 [07:21:45] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/158 [07:21:49] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/49 [07:21:54] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/158 [07:21:55] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/30 [07:21:57] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/27 [07:21:57] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/30 [07:22:03] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/22 [07:22:04] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/22 [07:22:04] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/23 [07:22:08] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/23 [07:23:06] 10Tools, 10Wikidata, 10wmde-wikidata-tech, 10Wikidata Dev Team (Wikidata.org Slice): [GENERAL] Deprecate connecting senses prototype - https://phabricator.wikimedia.org/T351829#9952778 (10ArthurTaylor) 05Stalled→03Open p:05Triage→03Medium a:03ArthurTaylor [07:24:53] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/23 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:24:56] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/23 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:24:59] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/23 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:25:22] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/22 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:25:27] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/22 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:25:29] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/22 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:25:54] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/30 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:25:57] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/30 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:26:00] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/30 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:26:16] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/158 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:26:19] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/158 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:26:20] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/158 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:26:46] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/22 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:26:59] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/27 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:27:02] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/27 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:27:06] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/27 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:28:57] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/78 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:28:59] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/78 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:29:01] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/78 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:29:18] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/50 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:29:20] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/50 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:29:23] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/50 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:29:47] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.28-20240704072720-8b68bcb7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/383 [07:32:09] (03update) 10dcaro: openapi_version_bump: add pre-commit hook [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/41 (https://phabricator.wikimedia.org/T356972) [07:32:21] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [cicd,infra] the pre-commit updater job fails trying to push to builds-admission - https://phabricator.wikimedia.org/T369188#9952822 (10dcaro) 05Open→03Resolved [07:44:16] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/49 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:44:19] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/49 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:44:19] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/49 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:44:54] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/99 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:44:57] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/99 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:45:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [07:45:31] (03open) 10sstefanova: dev: add kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/384 (https://phabricator.wikimedia.org/T329671) [07:45:41] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [07:45:46] (03update) 10sstefanova: dev: add kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/384 (https://phabricator.wikimedia.org/T329671) [07:49:57] (03update) 10dcaro: deployment: drop PSP reference [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T368142) (owner: 10aborrero) [07:53:51] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [07:54:03] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [07:54:23] (03update) 10sstefanova: dev: add kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/384 (https://phabricator.wikimedia.org/T329671) [07:55:42] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.108-20240704075431-1d17eb83 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/385 [07:55:57] FIRING: [8x] SystemdUnitDown: The service unit ceph-osd@16.service is in failed status on host cloudcephosd1009. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1009 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:58:03] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.313-20240704075512-a7a1394d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/386 [08:04:45] (03open) 10dcaro: toolforge_get_versions: show the mr for components too [repos/cloud/toolforge/lima-kilo] (show_package_origin_mr) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/159 [08:05:10] (03update) 10dcaro: toolforge_get_versions: show the mr for components too [repos/cloud/toolforge/lima-kilo] (show_package_origin_mr) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/159 [08:11:17] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9952880 (10Sebastian_Berlin-WMSE) There is now a release for CommonsExtension that works with OpenRefine 3.8: https://github.com/OpenRefine/CommonsExtension/releases/tag/v0.1.2. [08:14:27] (03approved) 10dcaro: api-gateway: bump to 0.0.28-20240704072720-8b68bcb7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/383 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:14:35] (03merge) 10dcaro: api-gateway: bump to 0.0.28-20240704072720-8b68bcb7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/383 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:14:36] (03update) 10dcaro: api-gateway: bump to 0.0.28-20240704072720-8b68bcb7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/383 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:22:26] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.1 Consulting Toolforge roots/maintainers - https://phabricator.wikimedia.org/T368601#9952927 (10dcaro) [08:23:46] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.2 Create "standard" tool to measure the number of steps for a deployment - https://phabricator.wikimedia.org/T368602#9952931 (10dcaro) [08:23:57] (03approved) 10aborrero: toolforge_get_versions: show the mr for components too [repos/cloud/toolforge/lima-kilo] (show_package_origin_mr) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/159 (owner: 10dcaro) [08:26:02] (03approved) 10aborrero: show package origin mr [repos/cloud/toolforge/lima-kilo] (version_with_colors) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/157 (owner: 10dcaro) [08:26:50] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.1 Consulting Toolforge roots/maintainers - https://phabricator.wikimedia.org/T368601#9952959 (10dcaro) [08:34:45] (03update) 10aborrero: dev: add kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/384 (https://phabricator.wikimedia.org/T329671) (owner: 10sstefanova) [08:34:46] (03approved) 10aborrero: dev: add kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/384 (https://phabricator.wikimedia.org/T329671) (owner: 10sstefanova) [08:35:20] (03approved) 10dcaro: deployment: drop PSP reference [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T368142) (owner: 10aborrero) [08:39:26] 06cloud-services-team, 10Technical-blog-posts: Tech blog post: "Wikimedia Toolforge: migrating Kubernetes from PodSecurityPolicy to kyverno" - https://phabricator.wikimedia.org/T368948#9952983 (10aborrero) 05Resolved→03Open hey @debt, some follow ups: * Could you please modify the post a bit, so the inclu... [08:45:33] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#9953006 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by root@cumin10... [08:52:08] 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [reimage,ceph] reimaging cloudcephosd hosts gets stuck in network configuration screen - https://phabricator.wikimedia.org/T369026#9953027 (10dcaro) cloudcephosd1009 was able to start installing after upgrading the nic firmware... [08:52:29] 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [reimage,ceph] reimaging cloudcephosd hosts gets stuck in network configuration screen - https://phabricator.wikimedia.org/T369026#9953030 (10dcaro) will do a couple more before declaring that the solution [09:05:05] (03update) 10dcaro: show package origin mr [repos/cloud/toolforge/lima-kilo] (version_with_colors) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/157 [09:05:25] (03update) 10dcaro: toolforge_get_versions: add some color when versions are weird [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/156 [09:06:00] (03update) 10dcaro: show package origin mr [repos/cloud/toolforge/lima-kilo] (version_with_colors) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/157 [09:06:09] (03update) 10dcaro: toolforge_get_versions: show the mr for components too [repos/cloud/toolforge/lima-kilo] (show_package_origin_mr) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/159 [09:06:13] (03update) 10dcaro: toolforge_get_versions: show the mr for components too [repos/cloud/toolforge/lima-kilo] (show_package_origin_mr) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/159 [09:06:23] (03merge) 10dcaro: toolforge_get_versions: add some color when versions are weird [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/156 [09:06:24] (03update) 10dcaro: show package origin mr [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/157 [09:09:34] (03update) 10sstefanova: dev: add kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/384 (https://phabricator.wikimedia.org/T329671) [09:10:14] (03merge) 10sstefanova: dev: add kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/384 (https://phabricator.wikimedia.org/T329671) [09:12:13] (03merge) 10dcaro: show package origin mr [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/157 [09:12:16] (03update) 10dcaro: toolforge_get_versions: show the mr for components too [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/159 [09:12:31] (03merge) 10dcaro: toolforge_get_versions: show the mr for components too [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/159 [09:13:19] (03update) 10dcaro: toolforge: add webservice configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/127 [09:18:05] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.2 Create "standard" tool to measure the number of steps for a deployment - https://phabricator.wikimedia.org/T368602#9953118 (10Slst2020) Some notes from brainstorming session with @dcaro: = Sample ap... [09:18:26] (03update) 10dcaro: toolforge: add webservice configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/127 [09:24:13] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, and 2 others: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#9953134 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by root@cumin10... [09:24:52] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [09:25:03] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [09:25:16] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, and 2 others: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#9953135 (10ops-monitoring-bot) Host rebooted by dcaro@cumin1002 with reason: upgraded packa... [09:28:32] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789) [09:28:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:28:38] T309789: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789 [09:31:33] (03update) 10dcaro: builds-builder: bump to 0.0.108-20240704075431-1d17eb83 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/385 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:32:01] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [09:32:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [09:34:17] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [09:34:28] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [09:34:33] 10Cloud-VPS (Debian Buster Deprecation), 10observability, 10Observability-Logging: Upgrade deployment-mwlog01.deployment-prep.eqiad1.wikimedia.cloud to bullseye or bookworm - https://phabricator.wikimedia.org/T369263#9953156 (10fgiunchedi) [09:35:28] 10Cloud-VPS (Debian Buster Deprecation), 10observability, 10Observability-Logging: Upgrade deployment-mwlog01.deployment-prep.eqiad1.wikimedia.cloud to bullseye or bookworm - https://phabricator.wikimedia.org/T369263#9953172 (10fgiunchedi) [09:35:29] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9953173 (10fgiunchedi) [09:36:24] (03approved) 10dcaro: builds-builder: bump to 0.0.108-20240704075431-1d17eb83 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/385 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:36:27] (03merge) 10dcaro: builds-builder: bump to 0.0.108-20240704075431-1d17eb83 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/385 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:36:27] (03update) 10dcaro: builds-builder: bump to 0.0.108-20240704075431-1d17eb83 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/385 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:37:25] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9953174 (10fgiunchedi) [09:38:42] (03update) 10dcaro: jobs-api: bump to 0.0.313-20240704075512-a7a1394d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/386 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:38:47] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [09:38:59] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [09:40:14] (03update) 10sstefanova: envvars-api: bump to 0.0.51-20240703124925-5829df80 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/382 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:43:44] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [09:43:55] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [09:45:21] (03open) 10sstefanova: dev: update dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/7 (https://phabricator.wikimedia.org/T329671) [09:46:14] (03update) 10dcaro: toolforge: add webservice configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/127 [09:57:12] (03merge) 10dcaro: jobs-api: bump to 0.0.313-20240704075512-a7a1394d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/386 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:57:15] (03update) 10dcaro: jobs-api: bump to 0.0.313-20240704075512-a7a1394d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/386 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:57:43] (03approved) 10dcaro: envvars-api: bump to 0.0.51-20240703124925-5829df80 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/382 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:57:49] (03update) 10dcaro: envvars-api: bump to 0.0.51-20240703124925-5829df80 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/382 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:00:51] 10Tool-spacemedia, 10Toolforge, 10video2commons: Enable SonarCloud usage for GitHub Toolforge projects - https://phabricator.wikimedia.org/T369267 (10Don-vip) 03NEW [10:01:45] 10Tool-spacemedia, 10Toolforge, 10video2commons: Enable SonarCloud usage for GitHub Toolforge projects - https://phabricator.wikimedia.org/T369267#9953281 (10Don-vip) [10:02:19] (03update) 10sstefanova: envvars-api: bump to 0.0.51-20240703124925-5829df80 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/382 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:06:24] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9953316 (10ABran-WMF) Ah @BTullis I see the issue you face, I had the same one, sorry for not... [10:10:26] (03update) 10dcaro: api: auth and proxy requests to the backend APIs [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/23 (https://phabricator.wikimedia.org/T363983) [10:10:28] (03update) 10dcaro: api: auth and proxy requests to the backend APIs [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/23 (https://phabricator.wikimedia.org/T363983) [10:12:49] (03merge) 10dcaro: api: auth and proxy requests to the backend APIs [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/23 (https://phabricator.wikimedia.org/T363983) [10:14:58] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.29-20240704101259-2cfd83b3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/387 (https://phabricator.wikimedia.org/T363983) [10:15:04] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [10:15:16] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [10:18:47] (03approved) 10sstefanova: openapi_version_bump: add pre-commit hook [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/41 (https://phabricator.wikimedia.org/T356972) (owner: 10dcaro) [10:19:24] (03update) 10dcaro: openapi_version_bump: add pre-commit hook [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/41 (https://phabricator.wikimedia.org/T356972) [10:19:24] (03merge) 10dcaro: openapi_version_bump: add pre-commit hook [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/41 (https://phabricator.wikimedia.org/T356972) [10:19:42] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:22:11] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [10:27:31] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [10:28:12] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [10:29:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:31:21] (03PS1) 10David Caro: ceph: don't fail for single ceph status failures [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052070 [10:31:21] (03PS1) 10David Caro: bootstrap_and_add: add cluster-wide silences [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052071 [10:31:21] (03PS1) 10David Caro: bootstrap_and_add: allow passing the batch size [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052072 [10:34:31] (03CR) 10CI reject: [V:04-1] ceph: don't fail for single ceph status failures [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052070 (owner: 10David Caro) [10:34:41] (03CR) 10CI reject: [V:04-1] bootstrap_and_add: add cluster-wide silences [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052071 (owner: 10David Caro) [10:34:42] (03CR) 10CI reject: [V:04-1] bootstrap_and_add: allow passing the batch size [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052072 (owner: 10David Caro) [11:36:37] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [11:36:49] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [11:37:38] (03open) 10aborrero: kubernetes: add some basic kyverno alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/17 (https://phabricator.wikimedia.org/T368515) [11:42:14] (03approved) 10dcaro: api-gateway: bump to 0.0.29-20240704101259-2cfd83b3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/387 (https://phabricator.wikimedia.org/T363983) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:42:17] (03update) 10dcaro: api-gateway: bump to 0.0.29-20240704101259-2cfd83b3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/387 (https://phabricator.wikimedia.org/T363983) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:42:18] (03merge) 10dcaro: api-gateway: bump to 0.0.29-20240704101259-2cfd83b3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/387 (https://phabricator.wikimedia.org/T363983) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:50:55] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Infrastructure Security: wikireplicas root access - https://phabricator.wikimedia.org/T344599#9953571 (10fnegri) a:03fnegri [11:59:34] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9953617 (10BTullis) OK, thanks. I've done that now. I had to go back and add the `~wmf1` suffi... [12:00:39] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [12:00:50] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [12:14:03] (03PS2) 10David Caro: bootstrap_and_add: allow passing the batch size [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052072 [12:14:03] (03PS1) 10David Caro: bootstrap_and_add: ignore osds that already were there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052094 [12:17:10] (03CR) 10CI reject: [V:04-1] bootstrap_and_add: allow passing the batch size [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052072 (owner: 10David Caro) [12:17:12] (03CR) 10CI reject: [V:04-1] bootstrap_and_add: ignore osds that already were there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1052094 (owner: 10David Caro) [12:23:53] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/14 [12:27:40] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9953675 (10ABran-WMF) Amazing 🎉 lets maye try the first deployment from a canary cumin host so... [12:43:10] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:10:20] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [toolforge,infra] Fix deprecated Kubelet flags - https://phabricator.wikimedia.org/T355881#9953777 (10aborrero) 05Open→03Resolved a:03aborrero [13:21:07] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9953812 (10BTullis) Thanks. Just checking, is `wmfdb-admin` installed by hand? I can't find an... [13:24:54] (03update) 10sstefanova: envvars-api: bump to 0.0.51-20240703124925-5829df80 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/382 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:27:07] (03approved) 10sstefanova: envvars-api: bump to 0.0.51-20240703124925-5829df80 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/382 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:27:15] (03merge) 10sstefanova: envvars-api: bump to 0.0.51-20240703124925-5829df80 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/382 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:44:18] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/14 [13:49:20] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 13Patch-For-Review: Remove namespace 666 from Wikitech - https://phabricator.wikimedia.org/T367254#9953960 (10Lucas_Werkmeister_WMDE) >>! In T367254#9892765, @taavi wrote: > That namespace does have deleted pages, can MW handle deleting a namespace with th... [14:10:40] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:11:25] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9954064 (10ABran-WMF) I'm affraid thats an answer I don't have @BTullis maybe @Marostegui or @... [14:16:09] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: [wmcs-backup] Backup snapshots of deleted volumes are never cleaned up - https://phabricator.wikimedia.org/T358774#9954092 (10fnegri) 05In progress→03Stalled [14:16:48] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: [wmcs-backup] Backup snapshots of deleted volumes are never cleaned up - https://phabricator.wikimedia.org/T358774#9954098 (10fnegri) Setting this to "Stalled" as I have too many things on my plate and this is not super urgent. I'd like to get back to it... [14:20:42] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:24:11] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9954163 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/437 [14:24:22] vivian-rook opened https://github.com/toolforge/paws/pull/437 [14:24:58] 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#9954162 (10fnegri) @Andrew @rook can you please review the PR at https://github.com/toolforge/superset-deploy/pull/26 and test if it works? @KCVelaga_... [14:30:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:34:49] 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#9954189 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/superset-deploy/pull/26 [14:35:02] vivian-rook closed https://github.com/toolforge/superset-deploy/pull/26 [14:35:11] 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#9954190 (10rook) >>! In T367393#9954161, @fnegri wrote: > @Andrew @rook can you please review the PR at https://github.com/toolforge/superset-deploy/pu... [14:35:22] FIRING: HAProxyBackendUnavailable: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:38:56] FIRING: [5x] SystemdUnitDown: The service unit systemd-journald-audit.socket is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:44:31] (03open) 10sstefanova: helpers: fix toolforge_get_versions.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/160 [14:52:47] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9954256 (10Marostegui) >>! In T368354#9953812, @BTullis wrote: > Thanks. > Just checking, is `... [14:53:56] RESOLVED: [5x] SystemdUnitDown: The service unit systemd-journald-audit.socket is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:54:31] (03update) 10sstefanova: dev: update dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/7 (https://phabricator.wikimedia.org/T329671) [14:54:39] PROBLEM - Host cloudcontrol1007 is DOWN: PING CRITICAL - Packet loss = 100% [14:55:10] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [14:55:12] 06cloud-services-team, 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic: [builds-cli,builds-api,builds-builder,jobs-cli,josb-api] user-story 14: I want have some certainty that the service is secure - https://phabricator.wikimedia.org/T325207#9954261 (10dcaro) We can r... [14:55:22] FIRING: [14x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:55:42] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9954262 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/437 [14:55:46] 10PAWS: update openrefine - https://phabricator.wikimedia.org/T363732#9954280 (10rook) 05Open→03Resolved [14:55:48] vivian-rook closed https://github.com/toolforge/paws/pull/437 [14:56:16] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9954263 (10rook) 05Open→03Resolved a:03rook [14:58:15] (03update) 10sstefanova: helpers: fix toolforge_get_versions.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/160 [14:58:47] FIRING: NodeDown: The node cloudcontrol1007 is unreachable. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [14:59:42] RECOVERY - Host cloudcontrol1007 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms [15:11:40] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [15:12:07] RESOLVED: [5x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:15:17] RESOLVED: NodeDown: The node cloudcontrol1007 is unreachable. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:28:21] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9954339 (10BTullis) OK, thanks @Marostegui I confirmed the hosts like this: ` btullis@cumin1... [15:35:29] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9954375 (10BTullis) Adding some debug logs, it's clear that it didn't add the `--defaults-grou... [15:40:53] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Toolsbeta: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T360699#9954384 (10dcaro) 05Open→03Resolved [15:44:23] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9954420 (10BTullis) Oh, somehow it's a bad build. The `mysql_cli.py` file doesn't contain my c... [15:58:14] 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#9954479 (10KCVelaga_WMF) @fnegri thank you! It is working as expected. I just created [[ https://superset.wmcloud.org/superset/dashboard/54/ | test das... [16:00:48] 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#9954483 (10fnegri) @KCVelaga_WMF thanks for testing! Yes it is kind of expected that all dbs are exposed, as in the SQL permissions are for all dbs end... [16:04:32] 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#9954490 (10KCVelaga_WMF) @fnegri yes, that's a good idea. [16:07:23] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): Upgrade Toolforge apt repository (tools-services hosts) to Debian Bullseye or later - https://phabricator.wikimedia.org/T311914#9954500 (10dcaro) [16:07:27] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): Upgrade Toolforge apt repository (tools-services hosts) to Debian Bullseye or later - https://phabricator.wikimedia.org/T311914#9954501 (10dcaro) p:05Triage→03High [16:08:06] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): Upgrade Toolforge apt repository (tools-services hosts) to Debian Bullseye or later - https://phabricator.wikimedia.org/T311914#9954499 (10dcaro) I think that would be a big change and we don't have much leeway before having to upgrade. I would be c... [16:09:55] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 12), 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#9954502 (10dcaro) p:05Triage→03High [16:10:33] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Modify db-mysql to connect to an-redacteddb1001 from cumin hosts - https://phabricator.wikimedia.org/T368354#9954504 (10BTullis) OK, I have fixed the build, deleted the botched packages from apt1002 and... [16:16:33] (03approved) 10dcaro: helpers: fix toolforge_get_versions.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/160 (owner: 10sstefanova) [16:16:45] (03update) 10dcaro: helpers: fix toolforge_get_versions.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/160 (owner: 10sstefanova) [16:17:17] (03update) 10dcaro: kubernetes: add some basic kyverno alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/17 (https://phabricator.wikimedia.org/T368515) (owner: 10aborrero) [16:22:31] (03update) 10sstefanova: helpers: fix toolforge_get_versions.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/160 [16:22:39] (03merge) 10sstefanova: helpers: fix toolforge_get_versions.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/160 [16:25:42] 06cloud-services-team, 10Cloud-VPS, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308 (10fnegri) 03NEW [16:27:34] 06cloud-services-team, 10Cloud-VPS, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#9954526 (10Marostegui) For us DBAs, we still have to apply production schema changes to its database (labtestwiki) [16:31:19] 06cloud-services-team, 10Cloud-VPS, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#9954536 (10fnegri) p:05Triage→03Low [17:04:12] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [17:04:16] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway [17:08:08] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [17:08:20] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [17:09:15] (03open) 10dcaro: api-gateway: fix apis urls [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/388 [17:09:39] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [17:09:50] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [17:11:27] (03approved) 10dcaro: api-gateway: fix apis urls [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/388 [17:11:31] (03merge) 10dcaro: api-gateway: fix apis urls [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/388 [17:11:32] (03update) 10dcaro: api-gateway: fix apis urls [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/388 [17:39:37] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9954702 (10Urbanecm_WMF) 05Open→03Stalled Hello @Don-vip, I can take a look at this, but it is currently not possible to download the images from Wikimedia... [17:52:14] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 07Design, 10Temporary accounts (Blockers to pilot wiki deployment): [Design EPIC] Global User Contributions - https://phabricator.wikimedia.org/T349901#9954728 (10Tchanders) [18:22:51] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Research: Replace or remove Debian Buster VMs in 'wmf-research-tools' cloud-vps project - https://phabricator.wikimedia.org/T367444#9954931 (10diego) > @diego: > * [[https://openstack-browser.toolforge.org/server/covid-data.wmf-research-tools.... [18:50:42] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:28:44] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9955016 (10Don-vip) Hi @Urbanecm_WMF ! It's weird, I can download files without problem from Cloud VPS: ` don-vip@worker-1:~/spacemedia$ hostname -A worker-1.s... [19:56:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [19:56:08] 06cloud-services-team: NovafullstackSustainedFailures The automated tests were unable to create, provision and decommission a VM in the last 5h - https://phabricator.wikimedia.org/T369316 (10phaultfinder) 03NEW [20:15:58] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789) [20:16:00] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789) [20:16:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:16:04] T309789: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789 [20:16:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:23:10] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [20:26:01] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789) [20:26:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:26:06] T309789: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789 [21:32:44] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [22:00:36] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [22:12:50] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [22:16:26] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [22:18:10] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [22:19:26] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [22:37:29] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 4 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [22:50:57] FIRING: CloudVPSDesignateLeaks: Detected 9 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:14:01] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [23:31:29] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [23:48:12] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [23:56:15] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures