[01:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:26:27] (03Abandoned) 10Abijeet Patro: Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1083800 (owner: 10L10n-bot) [03:28:11] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1085369 (owner: 10L10n-bot) [05:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:58:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:03:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:10:50] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10287429 (10Fnielsen) >>! In T368816#10286507, @Gopavasanth wrote: > @Soylacarli, @Fnielsen, @Nux, thank you for highlighting the bugs and gaps in the tool! The issues have been identified, f... [08:14:57] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:15:01] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:15:06] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:19:09] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.53-20241104081521-699c4926 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/566 [08:30:08] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:30:10] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:30:14] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:32:11] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/206 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:32:14] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/206 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:32:18] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/206 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:33:10] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/57 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:33:26] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.53-20241104083025-aa2f0400 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/567 [08:35:02] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T378676#10287475 (10Slst2020) This being a recurring maintenance task of sorts, should we maybe include it in clinic duty? [08:35:28] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:55:27] (03open) 10sstefanova: dev: update pre-commit to v4 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/48 [08:56:54] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/57 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:56:58] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/57 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:00:39] (03approved) 10dcaro: dev: update pre-commit to v4 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/48 (owner: 10sstefanova) [09:00:44] (03merge) 10dcaro: dev: update pre-commit to v4 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/48 (owner: 10sstefanova) [09:02:45] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.50-20241104090056-3af57c58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/568 [09:05:26] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:05:28] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:05:58] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:08:17] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:08:18] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:08:22] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/47 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:10:10] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.50-20241104090056-3af57c58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/568 [09:10:14] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.50-20241104090056-3af57c58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/568 [09:10:24] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.61-20241104090541-d0312d11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/569 [09:11:12] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/114 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:16:25] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [09:17:50] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/114 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:18:05] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/114 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:21:36] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [09:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:23:06] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/128 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:23:46] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [09:24:51] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.174-20241104091817-e3509011 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/570 [09:28:52] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [09:34:23] (03open) 10dcaro: api: fix merged version changing on every call [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/49 [09:36:54] (03open) 10sstefanova: dev: update pre-commit to v4 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/33 [09:39:46] (03update) 10sstefanova: api-gateway: bump to 0.0.50-20241104090056-3af57c58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/568 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:41:33] (03approved) 10sstefanova: api: fix merged version changing on every call [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/49 (owner: 10dcaro) [09:42:23] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [09:42:30] (03update) 10sstefanova: registry-admission: bump to 0.0.53-20241104081521-699c4926 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/566 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:47:11] 06cloud-services-team, 10Toolforge: Missing Perl packages on dev.toolforge.org for anomiebot workflows - https://phabricator.wikimedia.org/T360488#10287654 (10dcaro) @bd808 were you able to try it? I had a quick look at the container image repo, but it seems it's more than just one thing (some kind of templati... [09:47:36] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [09:48:27] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [09:49:19] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/128 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:49:24] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/128 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:50:46] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/14 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:51:12] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission [09:52:29] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.339-20241104094933-bb9cbca1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/571 [09:54:06] (03approved) 10sstefanova: api-gateway: bump to 0.0.50-20241104090056-3af57c58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/568 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:54:09] (03merge) 10sstefanova: api-gateway: bump to 0.0.50-20241104090056-3af57c58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/568 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:54:51] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/14 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:54:55] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/14 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:55:12] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [09:56:07] (03update) 10sstefanova: registry-admission: bump to 0.0.53-20241104081521-699c4926 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/566 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:56:54] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission [09:58:01] (03open) 10aborrero: projects: make some of the config options optional [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/113 [09:58:40] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-admission: bump to 0.0.20-20241104095506-bfd8b20d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/572 [09:59:07] (03merge) 10aborrero: projects: make some of the config options optional [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/113 [09:59:15] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [09:59:45] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:01:29] (03open) 10aborrero: projects_eqiad1: add wikipeoplestats project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/114 (https://phabricator.wikimedia.org/T378042) [10:03:47] (03merge) 10aborrero: projects_eqiad1: add wikipeoplestats project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/114 (https://phabricator.wikimedia.org/T378042) [10:03:54] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:05:30] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:06:00] (03CR) 10WMDE-Fisch: [C:03+1] "In the CLDR extension this path changed no. We need to apply that change here as well." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1082452 (https://phabricator.wikimedia.org/T377939) (owner: 10Awight) [10:08:12] (03update) 10dcaro: api: fix merged version changing on every call [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/49 [10:09:44] 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of WikiPeopleStats VPS project - https://phabricator.wikimedia.org/T378042#10287832 (10aborrero) 05Open→03Resolved [10:11:29] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [10:15:57] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [10:16:19] (03approved) 10sstefanova: registry-admission: bump to 0.0.53-20241104081521-699c4926 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/566 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:16:22] (03merge) 10sstefanova: registry-admission: bump to 0.0.53-20241104081521-699c4926 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/566 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:16:49] (03update) 10sstefanova: envvars-admission: bump to 0.0.20-20241104095506-bfd8b20d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/572 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:16:50] (03update) 10sstefanova: envvars-admission: bump to 0.0.20-20241104095506-bfd8b20d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/572 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:18:39] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:18:50] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/15 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:19:59] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [10:20:11] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: calico: bump to 0.0.15-20241104101859-e2c4ee9b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/573 [10:21:19] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:22:30] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:22:33] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:23:50] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: wmcs-k8s-metrics: bump to 0.0.21-20241104102245-a6f60a0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/574 [10:23:56] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/62 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:23:57] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/62 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:24:00] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/62 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:24:35] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:24:43] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [10:25:29] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.122-20241104102412-74466167 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/575 [10:25:31] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.122-20241104102412-74466167 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/575 [10:25:55] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [10:27:39] 10wikitech.wikimedia.org, 06SRE, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10287901 (10MatthewVernon) @jijiki can you expand on what you mean, please? This task is currently too broad... [10:28:41] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:28:45] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:29:58] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/37 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:30:01] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/37 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:30:05] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/37 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:31:26] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: volume-admission: bump to 0.0.58-20241104102855-3158d240 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/576 [10:32:29] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [10:33:54] (03approved) 10sstefanova: envvars-admission: bump to 0.0.20-20241104095506-bfd8b20d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/572 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:33:57] (03merge) 10sstefanova: envvars-admission: bump to 0.0.20-20241104095506-bfd8b20d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/572 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:34:25] (03update) 10sstefanova: volume-admission: bump to 0.0.58-20241104102855-3158d240 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/576 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:34:26] (03update) 10sstefanova: volume-admission: bump to 0.0.58-20241104102855-3158d240 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/576 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:35:40] (03open) 10dcaro: fix url [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/50 [10:37:25] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [10:38:30] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-idp-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:39:08] (03approved) 10sstefanova: fix url [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/50 (owner: 10dcaro) [10:43:20] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [10:50:35] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [10:52:25] (03update) 10dcaro: fix url [repos/cloud/toolforge/api-gateway] (fix_version) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/50 [10:56:20] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [11:02:25] (03approved) 10sstefanova: volume-admission: bump to 0.0.58-20241104102855-3158d240 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/576 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:02:28] (03merge) 10sstefanova: volume-admission: bump to 0.0.58-20241104102855-3158d240 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/576 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:02:49] (03update) 10sstefanova: ingress-admission: bump to 0.0.53-20241104083025-aa2f0400 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/567 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:02:50] (03update) 10sstefanova: ingress-admission: bump to 0.0.53-20241104083025-aa2f0400 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/567 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:05:51] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [11:11:18] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [11:13:17] (03update) 10dcaro: add token validation [repos/cloud/toolforge/components-api] (add_creation_date_to_token) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/32 (https://phabricator.wikimedia.org/T362066) [11:14:00] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [11:19:54] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [11:24:14] (03approved) 10sstefanova: ingress-admission: bump to 0.0.53-20241104083025-aa2f0400 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/567 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:24:17] (03merge) 10sstefanova: ingress-admission: bump to 0.0.53-20241104083025-aa2f0400 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/567 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:24:35] (03update) 10sstefanova: envvars-api: bump to 0.0.61-20241104090541-d0312d11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/569 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:24:37] (03update) 10sstefanova: envvars-api: bump to 0.0.61-20241104090541-d0312d11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/569 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:26:49] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 (T362867) [11:26:53] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [11:38:48] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 (T362867) [11:38:53] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [11:42:41] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 (T362867) [11:45:34] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [11:49:41] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 (T362867) [11:49:44] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [11:51:22] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [11:57:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 (T362867) [11:57:12] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [11:58:00] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [11:59:13] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api [12:03:20] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 (T362867) [12:03:25] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [12:06:50] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [12:11:44] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api [12:12:38] (03open) 10dcaro: auth: allow pass through for deploy urls with tokens [repos/cloud/toolforge/api-gateway] (fix_url) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/51 (https://phabricator.wikimedia.org/T362066) [12:13:20] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 (T362867) [12:13:24] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [12:16:52] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [12:22:26] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [12:22:43] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 (T362867) [12:22:46] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [12:24:12] (03approved) 10sstefanova: envvars-api: bump to 0.0.61-20241104090541-d0312d11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/569 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:24:16] (03merge) 10sstefanova: envvars-api: bump to 0.0.61-20241104090541-d0312d11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/569 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:25:02] (03update) 10dcaro: auth: allow pass through for deploy urls with tokens [repos/cloud/toolforge/api-gateway] (fix_url) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/51 (https://phabricator.wikimedia.org/T362066) [12:33:15] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 (T362867) [12:33:20] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [12:34:26] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 (T362867) [12:34:27] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 (T362867) [12:35:35] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 (T362867) [12:35:36] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 (T362867) [12:36:56] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 (T362867) [12:36:56] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 (T362867) [12:36:58] (03update) 10sstefanova: builds-api: bump to 0.0.174-20241104091817-e3509011 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/570 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:36:59] (03update) 10sstefanova: builds-api: bump to 0.0.174-20241104091817-e3509011 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/570 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:38:01] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [12:38:06] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 (T362867) [12:38:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 (T362867) [12:39:20] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 (T362867) [12:39:21] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 (T362867) [12:39:25] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [12:40:27] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 (T362867) [12:40:28] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 (T362867) [12:41:40] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 (T362867) [12:41:41] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [12:44:20] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [12:47:46] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [12:47:51] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [12:49:17] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [12:55:00] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [12:55:05] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:02:38] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [13:04:11] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [13:04:39] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [13:06:19] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [13:06:22] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:07:06] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [13:07:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 (T362867) [13:08:19] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 (T362867) [13:08:20] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 (T362867) [13:09:29] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 (T362867) [13:09:30] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 (T362867) [13:10:07] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [13:10:43] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 (T362867) [13:10:43] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 (T362867) [13:11:57] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 (T362867) [13:11:58] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 (T362867) [13:12:01] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:13:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 (T362867) [13:13:11] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 (T362867) [13:14:18] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 (T362867) [13:14:19] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 (T362867) [13:20:29] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 (T362867) [13:20:35] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:25:12] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 (T362867) [13:25:59] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 (T362867) [13:26:00] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 (T362867) [13:26:04] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:27:09] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 (T362867) [13:27:10] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 (T362867) [13:28:20] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 (T362867) [13:28:21] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 (T362867) [13:29:33] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 (T362867) [13:29:34] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 (T362867) [13:30:46] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 (T362867) [13:30:47] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 (T362867) [13:31:51] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 (T362867) [13:31:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 (T362867) [13:31:56] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:33:00] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 (T362867) [13:33:01] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 (T362867) [13:34:14] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 (T362867) [13:34:15] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 (T362867) [13:35:27] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 (T362867) [13:35:28] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 (T362867) [13:36:37] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 (T362867) [13:36:38] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 (T362867) [13:37:48] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 (T362867) [13:37:49] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 (T362867) [13:37:52] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:38:58] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 (T362867) [13:38:59] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 (T362867) [13:40:11] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 (T362867) [13:40:11] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 (T362867) [13:41:21] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 (T362867) [13:41:22] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 (T362867) [13:42:30] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 (T362867) [13:42:31] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 (T362867) [13:43:36] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 (T362867) [13:43:37] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 (T362867) [13:43:41] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:44:48] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 (T362867) [13:44:49] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 (T362867) [13:46:01] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 (T362867) [13:46:02] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 (T362867) [13:47:10] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 (T362867) [13:47:11] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 (T362867) [13:48:14] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10288577 (10aborrero) I'm now testing the code. * I disabled puppet on cloudlb2001-dev to force nova and designate request to go to cloudcontrol2004-dev * I disabled puppet on cloudco... [13:48:19] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 (T362867) [13:48:20] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 (T362867) [13:49:29] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 (T362867) [13:49:30] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 (T362867) [13:49:34] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:50:06] (03update) 10sstefanova: dev: update pre-commit to v4 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/33 [13:50:42] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 (T362867) [13:50:43] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 (T362867) [13:51:06] (03update) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/33 [13:51:56] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 (T362867) [13:51:57] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 (T362867) [13:52:39] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate deployment-poolcounter06.deployment-prep.eqiad.wmflabs is about to expire in 23d 23h 58m 30s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [13:53:11] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 (T362867) [13:53:12] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 (T362867) [13:54:22] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 (T362867) [13:54:23] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 (T362867) [13:55:30] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 (T362867) [13:55:31] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 (T362867) [13:55:36] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [13:56:41] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 (T362867) [13:56:42] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 (T362867) [13:57:51] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 (T362867) [13:57:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 (T362867) [13:59:02] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 (T362867) [13:59:03] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 (T362867) [14:00:13] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 (T362867) [14:00:14] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 (T362867) [14:01:24] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 (T362867) [14:01:25] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 (T362867) [14:01:29] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:02:32] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 (T362867) [14:02:33] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 (T362867) [14:08:29] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [14:08:29] !log aborrero@cloudcumin2001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [14:08:34] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 (T362867) [14:08:34] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [14:08:36] !log aborrero@cloudcumin2001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [14:08:41] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:09:33] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:09:36] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [14:10:56] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:10:59] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [14:13:26] 10Cloud-VPS (Project-requests), 10WMDE-TechWish-Survey, 07Unplanned-Sprint-Work, 03WMDE-TechWish-Sprint-2024-10-16: Request creation of wmde-techwishes-survey VPS project - https://phabricator.wikimedia.org/T378975 (10awight) 03NEW [14:13:58] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 (T362867) [14:14:02] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:14:43] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 (T362867) [14:14:44] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 (T362867) [14:20:15] 10Tool-lexeme-forms: Lexeme-forms on Toolforge returns error - https://phabricator.wikimedia.org/T378976 (10Fnielsen) 03NEW [14:20:44] 10Tool-lexeme-forms: Lexeme-forms on Toolforge returns error - https://phabricator.wikimedia.org/T378976#10288902 (10Fnielsen) [14:20:48] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 (T362867) [14:20:52] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:24:21] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 (T362867) [14:25:06] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 (T362867) [14:25:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 (T362867) [14:25:53] 10PAWS: Update build-and-push - https://phabricator.wikimedia.org/T378977 (10rook) 03NEW [14:26:05] 10Quarry: update build-and-push - https://phabricator.wikimedia.org/T378978 (10rook) 03NEW [14:26:17] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 (T362867) [14:26:18] 10PAWS: Update build-and-push - https://phabricator.wikimedia.org/T378977#10288932 (10rook) [14:26:18] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 (T362867) [14:26:21] 10PAWS, 10Quarry: update github action - https://phabricator.wikimedia.org/T348873#10288933 (10rook) [14:26:22] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:26:45] 10Quarry: update build-and-push - https://phabricator.wikimedia.org/T378978#10288936 (10rook) [14:26:50] 10PAWS, 10Quarry: update github action - https://phabricator.wikimedia.org/T348873#10288937 (10rook) [14:27:11] 10PAWS, 10Quarry: update github action - https://phabricator.wikimedia.org/T348873#10288939 (10rook) 05Open→03Resolved [14:27:30] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 (T362867) [14:27:31] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 (T362867) [14:28:37] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 (T362867) [14:28:38] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 (T362867) [14:29:45] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 (T362867) [14:29:46] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 (T362867) [14:30:55] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 (T362867) [14:30:56] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 (T362867) [14:32:04] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 (T362867) [14:32:05] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 (T362867) [14:32:08] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:33:12] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 (T362867) [14:33:13] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 (T362867) [14:33:14] 10PAWS: Update build-and-push - https://phabricator.wikimedia.org/T378977#10288965 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/460 [14:33:25] vivian-rook opened https://github.com/toolforge/paws/pull/460 [14:34:23] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 (T362867) [14:34:24] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 (T362867) [14:35:30] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 (T362867) [14:35:31] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 (T362867) [14:36:36] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 (T362867) [14:36:36] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 (T362867) [14:37:47] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 (T362867) [14:37:48] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 (T362867) [14:37:52] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:38:01] vivian-rook closed https://github.com/toolforge/paws/pull/460 [14:38:58] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 (T362867) [14:38:59] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 (T362867) [14:40:06] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 (T362867) [14:40:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 (T362867) [14:41:17] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 (T362867) [14:41:18] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 (T362867) [14:42:31] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 (T362867) [14:44:28] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 (T362867) [14:44:33] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [14:45:28] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 (T362867) [14:45:28] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 (T362867) [14:46:27] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 (T362867) [14:46:28] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 (T362867) [14:47:30] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 (T362867) [14:48:47] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/75 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:49:59] 10Toolforge (Toolforge iteration 16): [toolforge,grafana,infra] Grafana stopped showing the namespaces (and other stats) for toolforge namespaces - https://phabricator.wikimedia.org/T378981 (10dcaro) 03NEW [14:50:07] 10Toolforge (Toolforge iteration 16): [toolforge,grafana,infra] Grafana stopped showing the namespaces (and other stats) for toolforge namespaces - https://phabricator.wikimedia.org/T378981#10289040 (10dcaro) p:05Triage→03High [14:55:50] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Set up a bitu instance for codfw1dev - https://phabricator.wikimedia.org/T360795#10289061 (10SLyngshede-WMF) p:05Triage→03Low [14:56:07] 10Cloud-VPS (Project-requests), 10WMDE-TechWish-Survey, 07Unplanned-Sprint-Work, 03WMDE-TechWish-Sprint-2024-10-16: Request creation of wmde-techwishes-survey VPS project - https://phabricator.wikimedia.org/T378975#10289068 (10awight) [14:59:08] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:02:27] 10PAWS: Update build-and-push - https://phabricator.wikimedia.org/T378977#10289097 (10rook) 05Open→03Resolved [15:05:25] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [15:14:31] 10Cloud-VPS (Quota-requests): Increase catalyst storage to 550GB - https://phabricator.wikimedia.org/T378231#10289153 (10dcaro) a:03komla From my miscomment in the parent task: > LGTM, +1 > Let us know though if/when we can free some of it. [15:16:42] 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378841#10289161 (10dcaro) a:03komla This is a duplicate of {T378848} right? +1 [15:22:08] 10Cloud-VPS (Quota-requests): Increase catalyst storage to 550GB - https://phabricator.wikimedia.org/T378231#10289178 (10fnegri) +1 [15:24:12] 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378848#10289180 (10fnegri) +1 [15:29:25] 10Cloud-VPS (Quota-requests): Request floating IP for wikiwho project - https://phabricator.wikimedia.org/T376637#10289204 (10fnegri) Is this a valid use case for the new [Vanity Domains](https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet#Vanity_domains) feature? [15:44:08] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10289317 (10aborrero) Maybe we can create the project with the default quotas, and you can ask for whatever quota bump later once... [15:46:48] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10289347 (10Physikerwelt) >>! In T377655#10289317, @aborrero wrote: > Maybe we can create the project with the default quotas, an... [15:50:33] (03open) 10aborrero: eqiad1: create wikiqlever project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/115 (https://phabricator.wikimedia.org/T377655) [15:56:20] (03merge) 10aborrero: eqiad1: create wikiqlever project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/115 (https://phabricator.wikimedia.org/T377655) [15:56:27] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:57:27] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:59:32] 06cloud-services-team: openstack codfw1dev: API endpoints errors 2024-11-04 - https://phabricator.wikimedia.org/T378995 (10aborrero) 03NEW [16:00:14] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service, 13Patch-For-Review: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10289424 (10aborrero) 05Open→03Resolved a:03aborrero done, please create a separate ticket for qu... [16:00:46] 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378841#10289441 (10SDunlap) @dcaro yes, that one is a duplicate. I'm not sure how that happened. I'll mark the other one as invalid [16:01:17] 06cloud-services-team: openstack codfw1dev: API endpoints errors 2024-11-04 - https://phabricator.wikimedia.org/T378995#10289447 (10aborrero) 05Open→03In progress p:05Triage→03Medium [16:02:31] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [16:02:34] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [16:03:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [16:03:42] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868#10289459 (10fnegri) [16:03:43] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10289460 (10fnegri) [16:04:08] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:04:46] 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378841#10289452 (10SDunlap) 05Open→03Invalid Duplicate of T378848 [16:04:53] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [16:05:10] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [16:07:49] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868#10289465 (10fnegri) I removed {T362867} from the subtasks, to simplify the task graph and make the open subtasks more visible. [16:08:18] 10Cloud-VPS (Project-requests), 10WMDE-TechWish-Survey, 07Unplanned-Sprint-Work, 03WMDE-TechWish-Sprint-2024-10-16: Request creation of wmde-techwishes-survey VPS project - https://phabricator.wikimedia.org/T378975#10289481 (10awight) [16:08:27] 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378841#10289472 (10JJMC89) →14Duplicate dup:03T378848 [16:08:27] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [16:08:29] 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378848#10289474 (10JJMC89) [16:09:19] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:09:43] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:10:29] 06cloud-services-team, 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10289479 (10fnegri) Still happening after the 1.28 upgrade today: ` tools-bastion-12:~$ webservice shell Flag --wait has been deprecated, becau... [16:11:33] (03approved) 10sstefanova: builds-api: bump to 0.0.174-20241104091817-e3509011 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/570 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:11:33] (03merge) 10sstefanova: builds-api: bump to 0.0.174-20241104091817-e3509011 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/570 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:13:55] (03update) 10sstefanova: jobs-api: bump to 0.0.339-20241104094933-bb9cbca1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/571 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:13:57] (03update) 10sstefanova: jobs-api: bump to 0.0.339-20241104094933-bb9cbca1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/571 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:15:26] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:16:57] 10Tools, 10XTools: xtools erro - https://phabricator.wikimedia.org/T378998 (10WT20) 03NEW [16:17:54] 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#10289545 (10fnegri) I wonder if {T376637} could be one? Though they seem to want a redirect more than a real vanity url. [16:21:09] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:21:37] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:22:55] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [16:23:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [16:23:37] (03update) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/33 [16:25:02] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:30:21] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [16:31:13] 06cloud-services-team: openstack codfw1dev: API endpoints errors 2024-11-04 - https://phabricator.wikimedia.org/T378995#10289629 (10aborrero) 05In progress→03Resolved a:03aborrero as part of my work in {T378192} I had mistakenly disabled all nova-api backends: This was the diff when the puppet agent r... [16:34:37] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:35:43] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10289647 (10aborrero) [16:35:44] 06cloud-services-team, 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10289644 (10Slst2020) a:05LucasWerkmeister→03Slst2020 Yup, it's because I've (still) not deployed this. It's on my todo list now. [16:35:46] 06cloud-services-team: openstack codfw1dev: API endpoints errors 2024-11-04 - https://phabricator.wikimedia.org/T378995#10289648 (10aborrero) [16:35:49] 10Tool-video-answer-tool, 06Future-Audiences: Design: Add last modified and # of contributors metadata to video attribution - https://phabricator.wikimedia.org/T378383#10289655 (10Maryana) Jordan working on this – ETA next day or two hopefully! [16:36:36] (03CR) 10Majavah: [C:03+2] register: Also filter on user SUL ID [labs/striker] - 10https://gerrit.wikimedia.org/r/1076809 (https://phabricator.wikimedia.org/T359428) (owner: 10Majavah) [16:37:22] (03update) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/33 [16:38:50] (03Merged) 10jenkins-bot: register: Also filter on user SUL ID [labs/striker] - 10https://gerrit.wikimedia.org/r/1076809 (https://phabricator.wikimedia.org/T359428) (owner: 10Majavah) [16:39:49] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [16:41:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [16:44:28] 10Tool-toolwatch: Redesign Tool Details Navigation and Button Text - https://phabricator.wikimedia.org/T376173#10289686 (10Himacharanbatchu) 05In progress→03Resolved Looks like this is merged, Closing the ticket :) [16:45:39] 10Tool-toolwatch, 06Toolforge-standards-committee, 07Privacy: toolwatch loads third party resources - https://phabricator.wikimedia.org/T378901#10289683 (10Himacharanbatchu) a:03Himacharanbatchu Thanks @Taavi for the nice observations you made, I'll fix this :) [16:46:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [16:48:25] (03approved) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/33 [16:48:29] (03update) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/33 [16:48:30] (03merge) 10sstefanova: dev: manage pre-commit in tox (not poetry) [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/33 [16:49:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [16:53:16] 10wikitech.wikimedia.org, 06SRE, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10289712 (10bd808) I can't speak for Effie, but my imagined reimplementation of wikitech-static would be to produce an OCI container image daily containing Wikitech's content along with a Medi... [16:54:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [16:57:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [16:59:07] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10289757 (10aborrero) I can't make the service work as expected. It seems `designate-sink` refuses to call the handlers, with this config: ` [service:sink] enabled_notification_handle... [16:59:28] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10289758 (10bd808) >>! In T368816#10287429, @Fnielsen wrote: > It has not improved. Go to https://tool-watch.toolforge.org/search?search=ordia and see that it says Unavailable. The URL is wro... [17:02:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [17:06:13] 10Cloud-VPS (Project-requests): Request creation of capacity-exchange VPS project - https://phabricator.wikimedia.org/T296556#10289790 (10joanna_borun) @Andrew maybe worth checking with https://meta.wikimedia.org/wiki/User:AJurno_(WMB) , https://meta.wikimedia.org/wiki/User:AInacio_(WMB) [17:15:29] 10wikitech.wikimedia.org: Can't log into Wikitech to unify - https://phabricator.wikimedia.org/T378897#10289881 (10bd808) →14Duplicate dup:03T376267 [17:16:14] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10289878 (10bd808) [17:20:52] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10289907 (10bd808) >>! In T378897#10286892, @Miniapolis wrote: >|**Wikitech account/LDAP:**| Miniapolis| >|**SUL account**| Miniapolis| >|**Account linked on [[ https://idm.wikimedia.org/ | I... [17:21:24] (03update) 10dcaro: auth: allow pass through for deploy urls with tokens [repos/cloud/toolforge/api-gateway] (fix_url) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/51 (https://phabricator.wikimedia.org/T362066) [17:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:26:54] 10Tool-lexeme-forms: Lexeme-forms on Toolforge returns error - https://phabricator.wikimedia.org/T378976#10289964 (10LucasWerkmeister) No idea what’s wrong: `lang=shell-session tools.lexeme-forms@tools-bastion-13:~$ kubectl get pods NAME READY STATUS RESTARTS AGE lexeme-f... [17:31:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [17:31:26] 10Tool-lexeme-forms: Lexeme-forms on Toolforge returns error - https://phabricator.wikimedia.org/T378976#10290010 (10LucasWerkmeister) The pod didn’t log anything else before it died, apparently: `lang=shell-session tools.lexeme-forms@tools-bastion-13:~$ kubectl logs --follow lexeme-forms-7cdc74df65-94lmb ++ wh... [17:32:49] 10Tool-lexeme-forms: Lexeme-forms on Toolforge returns error - https://phabricator.wikimedia.org/T378976#10290015 (10LucasWerkmeister) Looks like that worked 🤷 [17:34:49] 10Tool-lexeme-forms: Lexeme-forms on Toolforge returns error - https://phabricator.wikimedia.org/T378976#10290016 (10LucasWerkmeister) 05Open→03Resolved a:03LucasWerkmeister [17:35:40] 10Tool-lexeme-forms: Lexeme-forms on Toolforge returns error - https://phabricator.wikimedia.org/T378976#10290028 (10Fnielsen) A restart helped and it is also working from Ordia now. Thanks. [17:36:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [17:38:32] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10290022 (10LucasWerkmeister) In {T378976}, a webservice was dead for a few hours until I manually stopped and started it; might be related to this u... [17:40:34] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10290055 (10Miniapolis) Please close my ticket or post or whatever, since I was finally able to change my password and link the Wikitech account to my SUL. Thanks and all the best, Miniapolis [17:43:03] (03open) 10dcaro: toolforge_deploy_mr: added restart of deployments if no diff in chart [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/207 [17:53:20] (03update) 10dcaro: add token validation [repos/cloud/toolforge/components-api] (add_creation_date_to_token) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/32 (https://phabricator.wikimedia.org/T362066) [18:00:37] (03update) 10dcaro: add token validation [repos/cloud/toolforge/components-api] (add_creation_date_to_token) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/32 (https://phabricator.wikimedia.org/T362066) [18:03:47] (03open) 10jjmc89: update wikimedia-hackathon config [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/10 [18:04:12] (03update) 10dcaro: add token validation [repos/cloud/toolforge/components-api] (add_creation_date_to_token) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/32 (https://phabricator.wikimedia.org/T362066) [18:04:13] (03update) 10jjmc89: update wikimedia-hackathon config [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/10 [18:04:54] (03update) 10jjmc89: update wikimedia-hackathon config [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/10 [18:08:52] (03update) 10jjmc89: update wikimedia-hackathon config [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/10 [18:11:03] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10290167 (10Fnielsen) I am not sure what that means. Is it something that I should do? https://tool-watch.toolforge.org/tools/1117 still shows unavailability. [18:16:07] (03update) 10dcaro: add token validation [repos/cloud/toolforge/components-api] (add_creation_date_to_token) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/32 (https://phabricator.wikimedia.org/T362066) [18:19:32] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Image layout adjustment for lower-res images - https://phabricator.wikimedia.org/T375690#10290230 (10Maryana) When do we switch? Do we need UI to switch? Where does the text get positioned relative to image? [18:21:10] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Image layout adjustment for lower-res images - https://phabricator.wikimedia.org/T375690#10290247 (10Maryana) [18:21:43] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Image layout adjustment for lower-res images - https://phabricator.wikimedia.org/T375690#10290252 (10Maryana) [18:22:14] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: [Spike] Image layout adjustment for lower-res images - https://phabricator.wikimedia.org/T375690#10290256 (10Maryana) a:03etz [18:23:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:28:06] RESOLVED: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:52:00] (03merge) 10bd808: update wikimedia-hackathon config [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/10 (owner: 10jjmc89) [18:52:03] 06cloud-services-team, 10Toolforge: Missing Perl packages on dev.toolforge.org for anomiebot workflows - https://phabricator.wikimedia.org/T360488#10290350 (10bd808) >>! In T360488#10287654, @dcaro wrote: > @bd808 were you able to try it? I have not yet made enough time to try adding wrapper scripts for the P... [19:13:09] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10290425 (10bd808) >>! In T368816#10290167, @Fnielsen wrote: > I am not sure what that means. Is it something that I should do? https://tool-watch.toolforge.org/tools/1117 still shows unavail... [19:22:40] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10290458 (10Fnielsen) Thanks. I did not seem to have permission to edit the "Tool type" field. [21:08:19] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10290886 (10Nux) I've added `"tool_type": "web app"` to my `toolinfo.json`. That seems to have fixed toolhub too. Example for those that want to fix in JSON: https://authors.toolforge.org/to... [21:18:12] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:22:37] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:57:33] 06cloud-services-team, 10Toolforge: Toolforge job fails to find library installed via aptfile - https://phabricator.wikimedia.org/T373251#10290992 (10bd808) The Aptfile installer does things that are "not normal" when it installs packages. For reasons that are not helpful to explain here, packages are installe... [22:28:02] 06cloud-services-team, 10Cloud-VPS, 10Striker: OpenStack project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030 (10bd808) 03NEW [22:33:42] 06cloud-services-team, 10Cloud-VPS, 10Striker: OpenStack project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10291067 (10bd808) [22:33:44] 06cloud-services-team, 10Cloud-VPS, 07Epic: Wind down use of project ID and project name equivalency in OpenStack - https://phabricator.wikimedia.org/T274268#10291068 (10bd808) [22:34:13] 06cloud-services-team, 10Cloud-VPS, 10Striker: OpenStack project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10291066 (10bd808) https://openstack-browser.toolforge.org/project/5238620953c040e7a9effbe47d4e0932 is the UUID named page for the "wikiqlev... [23:14:04] 06Toolforge-standards-committee, 07User-notice: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10291128 (10bd808) [23:14:50] 06Toolforge-standards-committee, 07User-notice: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10291129 (10bd808) New members "officially" announced on cloud-announce: https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/KK3HMB4K... [23:27:55] 06Toolforge-standards-committee, 10Tools, 07Privacy: activity.toolforge.org loads third-party content for d3 - https://phabricator.wikimedia.org/T311297#10291163 (10LucasWerkmeister) 05Open→03Invalid Tool was apparently deleted / archived almost two years ago, nothing more to be done here: `lang=shel... [23:41:34] 06Toolforge-standards-committee, 10Humaniki, 07Privacy: humaniki.wmcloud.org loads 3rd party content - https://phabricator.wikimedia.org/T299237#10291237 (10LucasWerkmeister) More to the point, plausible.io is an analytics service. While I commend the Humaniki team on not using Google Analytics, I’m still no... [23:45:28] 06Toolforge-standards-committee, 10Tools, 06Privacy Engineering, 07Privacy: Tool "blockyquery" loads resources from cloudflare and jquery.com - https://phabricator.wikimedia.org/T242668#10291251 (10LucasWerkmeister) 05Open→03Stalled The tool is currently offline, having been disabled as part of the [gr...