[00:13:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [01:18:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [02:43:24] (03open) 10bd808: Draft: Build service integration [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/3 [03:04:03] (03update) 10bd808: Draft: Build service integration [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/3 (https://phabricator.wikimedia.org/T370565) [03:12:06] (03close) 10bd808: Draft: Build service integration [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/3 (https://phabricator.wikimedia.org/T370565) [03:12:56] (03open) 10bd808: Add wrapper script for buildservice image [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/4 (https://phabricator.wikimedia.org/T370565) [03:15:04] (03merge) 10bd808: Add wrapper script for buildservice image [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/4 (https://phabricator.wikimedia.org/T370565) [03:38:08] (03open) 10bd808: Switch to buildservice [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/5 (https://phabricator.wikimedia.org/T370565) [03:42:01] (03merge) 10bd808: add new maintainers - bd808 and JJMC89 [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/4 (owner: 10jjmc89) [03:42:15] (03update) 10bd808: Switch to buildservice [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/5 (https://phabricator.wikimedia.org/T370565) [03:46:03] (03merge) 10bd808: Switch to buildservice [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/5 (https://phabricator.wikimedia.org/T370565) [03:49:14] 10Toolforge: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610 (10tstarling) 03NEW [04:14:08] 10Toolforge: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10001262 (10Legoktm) > Given multiple configuration sources, ImageMagick always uses the lowest specified limit. So setting a higher limit by adding a configuration file,... [05:22:53] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10001296 (10Slst2020) [06:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:26:37] 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10001340 (10Slst2020) [06:26:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10001342 (10Slst2020) 05Open→03In progress [06:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:37:39] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:40:15] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10001348 (10ayounsi) There is an outstanding diff on the switch for `cloudcephmon1006`. It looks correct, but could DCops double check it and make sure th... [06:42:39] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:28:25] (03open) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069) [08:28:33] (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069) [08:44:10] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [08:44:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:52:19] (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069) [08:53:07] (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069) [08:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:09:33] (03update) 10sstefanova: metrics: initialize all the stats to 0 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/109 (owner: 10dcaro) [09:09:34] (03approved) 10sstefanova: metrics: initialize all the stats to 0 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/109 (owner: 10dcaro) [09:11:28] (03update) 10sstefanova: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) (owner: 10dcaro) [09:14:37] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [09:14:39] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [09:14:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:14:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:23:58] (03merge) 10dcaro: metrics: initialize all the stats to 0 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/109 [09:24:04] (03update) 10dcaro: cli: add namespaces and host to the user agent [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/52 [09:25:27] (03merge) 10dcaro: cli: add namespaces and host to the user agent [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/52 [09:26:13] (03update) 10dcaro: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) [09:26:58] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.322-20240722092409-4e3e1e7b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/435 [09:27:07] (03update) 10dcaro: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) [09:32:14] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [09:32:25] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [09:38:26] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge, 07Epic: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#10001663 (10joanna_borun) [09:38:56] (03open) 10dcaro: run_functional_tests: update the toolforge-deploy repo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/436 [09:42:19] (03approved) 10sstefanova: run_functional_tests: update the toolforge-deploy repo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/436 (owner: 10dcaro) [09:42:55] 10Toolforge, 13Patch-For-Review: [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with build-only features - https://phabricator.wikimedia.org/T362069#10001675 (10Slst2020) we might want to use something like this for prometheus: https://github.com/trallnag/prometheus-f... [09:43:11] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [09:43:22] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [09:43:42] (03merge) 10dcaro: run_functional_tests: update the toolforge-deploy repo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/436 [09:43:44] (03update) 10dcaro: run_functional_tests: update the toolforge-deploy repo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/436 [09:58:49] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [09:59:01] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [10:00:39] (03approved) 10dcaro: jobs-api: bump to 0.0.322-20240722092409-4e3e1e7b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/435 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:00:44] (03update) 10dcaro: jobs-api: bump to 0.0.322-20240722092409-4e3e1e7b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/435 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:01:11] (03merge) 10dcaro: jobs-api: bump to 0.0.322-20240722092409-4e3e1e7b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/435 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:01:24] (03open) 10sstefanova: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069) [10:01:30] (03update) 10sstefanova: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069) [10:01:47] (03PS28) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [10:02:12] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, 05Goal: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#10001795 (10joanna_borun) [10:02:58] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 13), 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#10001797 (10joanna_borun) [10:03:15] (03open) 10dcaro: d/changelog: bump to 0.0.10 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/54 (https://phabricator.wikimedia.org/T370393) [10:03:45] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10001800 (10joanna_borun) [10:04:31] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10001802 (10joanna_borun) [10:04:48] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10Data-Services, 05Goal: Improve trove backup/restore - https://phabricator.wikimedia.org/T356291#10001803 (10joanna_borun) [10:05:18] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 07Epic: Migrate largest ToolsDB users to Trove - https://phabricator.wikimedia.org/T291782#10001805 (10joanna_borun) [10:05:39] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: [toolsdb] test failover procedure - https://phabricator.wikimedia.org/T344719#10001806 (10joanna_borun) [10:06:01] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge, 05Goal: [harbor] Create backups and/or replication - https://phabricator.wikimedia.org/T336668#10001807 (10joanna_borun) [10:06:27] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Upgrade cloud-vps openstack to version 'Caracal' - https://phabricator.wikimedia.org/T369044#10001809 (10joanna_borun) [10:09:30] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [trove] define process for updating docker images - https://phabricator.wikimedia.org/T359534#10001811 (10joanna_borun) [10:10:00] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge, 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10001812 (10joanna_borun) [10:22:25] (03update) 10sstefanova: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) (owner: 10dcaro) [10:22:27] (03approved) 10sstefanova: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) (owner: 10dcaro) [10:23:52] (03open) 10dcaro: toolforge_get_versions: warm up cache properly [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/438 [10:35:22] (03approved) 10dcaro: d/changelog: bump to 0.0.10 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/54 (https://phabricator.wikimedia.org/T370393) [10:35:26] (03merge) 10dcaro: d/changelog: bump to 0.0.10 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/54 (https://phabricator.wikimedia.org/T370393) [10:38:10] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [10:41:04] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29687 bytes in 1.667 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [10:42:15] (03approved) 10dcaro: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046) (owner: 10sstefanova) [10:52:59] (03PS29) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [10:55:49] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [11:07:52] (03PS30) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:10:59] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [11:20:09] (03PS31) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:23:19] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [11:24:14] (03PS32) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:25:29] (03PS1) 10Jaime Nuche: jobs.yaml: run `update.php` every 15 minutes [labs/tools/train-blockers] - 10https://gerrit.wikimedia.org/r/1055894 [11:27:46] (03CR) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [11:29:33] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [11:29:42] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [11:31:09] (03PS33) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:31:11] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [11:31:31] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [11:34:39] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [11:34:54] (03PS34) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:34:54] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [11:35:04] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [11:35:20] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [11:36:45] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [11:36:56] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [11:38:20] (03PS35) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:39:47] (03PS36) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:39:52] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [11:39:59] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [11:40:58] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:41:35] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [11:41:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:41:43] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:41:47] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [11:41:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:48:20] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge: [components-api] Get a minimal version of the config with build-only data - https://phabricator.wikimedia.org/T362070#10002019 (10joanna_borun) [11:48:43] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge, 13Patch-For-Review: [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with build-only features - https://phabricator.wikimedia.org/T362069#10002022 (10joanna_borun) [11:49:47] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge: [components-api] Develop the webhook mechanism to trigger a deployment - https://phabricator.wikimedia.org/T362066#10002023 (10joanna_borun) [11:50:03] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge: [components-api] add one-off, scheduled and continuous jobs support to the yaml + api - https://phabricator.wikimedia.org/T362075#10002024 (10joanna_borun) [11:51:28] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#10002026 (10joanna_borun) [12:12:47] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [12:12:49] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [12:12:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:12:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:24:18] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/16 [12:25:27] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1055914 (owner: 10L10n-bot) [12:25:29] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1055916 (owner: 10L10n-bot) [12:30:57] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [12:31:32] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:31:36] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:35:08] (03PS37) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [12:35:11] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:35:19] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:41:12] (03PS38) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [12:43:01] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [12:43:04] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:43:17] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:44:20] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [12:45:01] (03PS39) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [12:47:35] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [12:47:40] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:47:48] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:48:49] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [12:48:57] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:49:11] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:55:24] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:55:24] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [12:56:01] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [12:59:53] (03update) 10sstefanova: toolforge_get_versions: warm up cache properly [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/438 (owner: 10dcaro) [12:59:53] (03approved) 10sstefanova: toolforge_get_versions: warm up cache properly [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/438 (owner: 10dcaro) [12:59:59] (03merge) 10dcaro: toolforge_get_versions: warm up cache properly [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/438 [13:04:25] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:04:35] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:04:41] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:04:43] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [13:04:50] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:05:41] (03update) 10dcaro: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) (owner: 10raymond-ndibe) [13:06:32] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [13:06:34] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:06:56] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:09:52] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [13:09:55] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:10:25] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:11:19] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [13:11:21] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:11:45] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:14:10] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [13:14:14] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:14:47] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:28:36] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [13:28:52] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [13:32:15] (03open) 10sstefanova: components: add components-api [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/173 (https://phabricator.wikimedia.org/T362069) [13:33:34] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:34:18] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [13:35:13] (03CR) 10FNegri: [C:03+1] "I tested both plan and apply with test-cookbook and it seems to work nicely!" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [13:40:26] (03CR) 10FNegri: [C:03+1] cookbooks: add new wmcs.openstack.tofu cookbook (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [13:41:19] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:41:40] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [13:47:07] (03CR) 10FNegri: [C:03+1] cookbooks: add new wmcs.openstack.tofu cookbook (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [13:48:25] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10002393 (10Ameisenigel) It seems like there is some progress on this task: https://wikitech.wikimedia.org/wiki/News/Migrating_Wikitech_Account_to_SUL [13:51:02] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge, 05Goal: [harbor] Move harbor data to object storage service - https://phabricator.wikimedia.org/T350687#10002404 (10dcaro) a:03Raymond_Ndibe [13:51:41] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge, 05Goal: [harbor] Create backups and/or replication - https://phabricator.wikimedia.org/T336668#10002405 (10dcaro) a:03Raymond_Ndibe [13:53:03] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1055916 (owner: 10L10n-bot) [13:56:54] (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069) [13:58:27] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1055914 (owner: 10L10n-bot) [14:00:57] (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069) [14:01:10] (03update) 10sstefanova: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069) [14:01:39] (03update) 10sstefanova: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069) [14:02:39] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [14:02:49] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [14:03:27] (03approved) 10fnegri: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) (owner: 10aborrero) [14:03:49] (03PS1) 10Sebastian Berlin (WMSE): Use tasks for updating images when a campaign is updated [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1055949 [14:03:49] (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069) [14:04:02] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [14:04:20] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [14:04:42] !log andrew@cloudcumin1001 deployment-prep START - Cookbook wmcs.openstack.rebuild_dbinstance [14:06:40] !log andrew@cloudcumin1001 deployment-prep END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) [14:08:21] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723#10002475 (10Andrew) [14:08:40] (03update) 10sstefanova: build: upgrade k8s dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T370046) [14:08:55] !log andrew@cloudcumin1001 deployment-prep START - Cookbook wmcs.openstack.rebuild_dbinstance [14:10:23] (03update) 10sstefanova: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046) [14:10:25] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723#10002478 (10Andrew) >>! In T369723#9989025, @Audiodude wrote: > It would be best for mwoffliner (which runs the WP 1.0 Bot) if the maintenance wasn't between... [14:10:27] (03CR) 10David Caro: "Some notes there, nice runbook" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [14:10:51] !log andrew@cloudcumin1001 deployment-prep END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) [14:11:20] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723#10002481 (10Andrew) [14:11:38] (03CR) 10David Caro: cookbooks: add new wmcs.openstack.tofu cookbook (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [14:13:46] (03update) 10sstefanova: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046) [14:14:33] (03merge) 10dcaro: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) [14:14:42] 10wikitech.wikimedia.org: Remove "renameuser" permission from the "bureaucrat" group on Wikitech - https://phabricator.wikimedia.org/T370638 (10GTrang) 03NEW [14:19:30] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.57-20240722141446-921d24d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/439 (https://phabricator.wikimedia.org/T367181) [14:21:16] 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10002542 (10Slst2020) [14:22:01] 10wikitech.wikimedia.org, 10Gerrit, 07LDAP: Rename account Zoranzoki21 to Kizule on Gerrit - https://phabricator.wikimedia.org/T260647#10002538 (10GTrang) 05Open→03Declined Per the note at the top of https://wikitech.wikimedia.org/wiki/SRE/LDAP/Renaming_users, we no longer rename LDAP accounts. [14:24:20] (03update) 10sstefanova: build: upgrade k8s dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/10 (https://phabricator.wikimedia.org/T370046) [14:34:27] 10Tool-spacemedia, 10Toolforge (Toolforge iteration 13), 10video2commons: Enable SonarCloud usage for GitHub Toolforge projects - https://phabricator.wikimedia.org/T369267#10002632 (10dcaro) >>! In T369267#9999579, @Don-vip wrote: > Thank you @dcaro! I managed to perform an analysis on spacemedia: > - https:... [14:39:00] 10wikitech.wikimedia.org, 07LDAP: Rename LDAP account to "Chlod" - https://phabricator.wikimedia.org/T279927#10002654 (10GTrang) 05Open→03Declined Per the note at the top of https://wikitech.wikimedia.org/wiki/SRE/LDAP/Renaming_users, we no longer rename LDAP accounts. [14:40:13] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [14:40:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:40:25] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [14:40:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:40:52] 10wikitech.wikimedia.org, 07LDAP: Merge developer accounts for Riley Huntley - https://phabricator.wikimedia.org/T238893#10002659 (10GTrang) 05Open→03Invalid Merging Wikitech accounts is not technically possible. [14:46:15] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#10002696 (10Andrew) Arturo suggests that we just do everything as root and abandon gitpuppet entirely. [14:47:32] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#10002715 (10Andrew) Also david mentioned that we have 'pgit' which is an alias for using the gitpuppet user. We can either lea... [14:48:23] (03update) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [14:50:15] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10002720 (10Aklapper) @Komla: Could you provide some additional info (as you created that wiki page)? Who's the main assignee? TIA [14:57:42] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace or delete deployment-mediawiki[11-12].deployement-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361387#10002755 (10Southparkfan) 05Open→03Resolved [15:00:26] !log andrew@cloudcumin1001 deployment-prep START - Cookbook wmcs.openstack.rebuild_dbinstance [15:01:02] (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) [15:01:35] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-urldownloader03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370466#10002770 (10Southparkfan) 05Open→03R... [15:02:24] !log andrew@cloudcumin1001 deployment-prep END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) [15:02:56] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723#10002788 (10Andrew) [15:10:45] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [15:10:47] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [15:10:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:10:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:10:59] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10002864 (10Papaul) @Jclark-ctr ok let me look [15:30:54] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-push-notifications01.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370459#10002978 (10Southparkfan) 05Open... [15:31:05] (03PS40) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:31:13] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [15:31:14] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [15:34:09] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [15:35:31] (03PS41) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:35:34] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [15:35:54] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [15:36:09] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: introduce additional gitlab-ci automation - https://phabricator.wikimedia.org/T370652#10003026 (10fnegri) [15:39:28] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [15:39:38] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [15:39:44] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] cookbooks: add new wmcs.openstack.tofu cookbook (037 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [15:41:23] 06cloud-services-team, 10Cloud-VPS, 07Epic, 13Patch-For-Review: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#10003084 (10aborrero) Next cookbook patch: * git checkout the gitlab MR into a different directory * avoid touching / rebasing at all the main /... [15:41:43] vivian-rook closed https://github.com/toolforge/paws/pull/438 [15:42:09] 10PAWS: jupyterlab to 4.2.4 - https://phabricator.wikimedia.org/T370658 (10rook) 03NEW [15:42:53] vivian-rook opened https://github.com/toolforge/paws/pull/442 [15:42:55] 10PAWS: jupyterlab to 4.2.4 - https://phabricator.wikimedia.org/T370658#10003132 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/442 [15:43:48] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461#10003140 (10Southparkfan) Had to delete sessionstorage05 (bookworm... [15:46:20] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [15:46:51] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [15:46:58] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [15:47:31] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 [15:48:41] FIRING: CloudVPSDesignateLeaks: Detected 10 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:50:30] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: investigate S3 spurious endpoint errors - https://phabricator.wikimedia.org/T370660 (10aborrero) 03NEW [15:50:31] (03merge) 10aborrero: tofu-infra: import codfw1dev subnets [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17 (https://phabricator.wikimedia.org/T370037) [15:50:31] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:51:06] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:03:22] 06cloud-services-team, 10Toolforge (Toolforge iteration 13): Upgrade Toolforge apt repository (tools-services hosts) to Debian Bullseye or later - https://phabricator.wikimedia.org/T311914#10003306 (10dcaro) a:03dcaro [16:04:27] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461#10003315 (10Southparkfan) Puppet fails to install the Cassandra in... [16:04:44] 06cloud-services-team, 10Toolforge (Toolforge iteration 13): Upgrade Toolforge apt repository (tools-services hosts) to Debian Bullseye or later - https://phabricator.wikimedia.org/T311914#10003311 (10dcaro) 05Open→03In progress [16:05:34] (03open) 10aborrero: data/: introduce eqiad1-r network and subnet information [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21 (https://phabricator.wikimedia.org/T370037) [16:05:43] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21 [16:05:58] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21 [16:05:59] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#10003326 (10Southparkfan) [16:07:43] (03update) 10aborrero: data/: introduce eqiad1-r network and subnet information [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21 (https://phabricator.wikimedia.org/T370037) [16:07:51] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21 [16:08:09] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21 [16:14:29] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-services-06 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:16:10] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [16:19:44] (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) [16:19:45] (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) [16:21:15] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [16:22:09] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [16:22:22] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [16:24:18] vivian-rook closed https://github.com/toolforge/paws/pull/442 [16:24:29] 10PAWS: jupyterlab to 4.2.4 - https://phabricator.wikimedia.org/T370658#10003472 (10rook) 05Open→03Resolved [16:24:29] 10PAWS: jupyterlab to 4.2.4 - https://phabricator.wikimedia.org/T370658#10003473 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/442 [16:27:53] (03update) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 [16:27:57] (03update) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 [16:29:43] (03update) 10raymond-ndibe: [builds-cli] bug fix [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/79 [16:30:16] (03approved) 10raymond-ndibe: [builds-cli] bug fix [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/79 [16:30:25] (03unapproved) 10raymond-ndibe: [builds-cli] bug fix [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/79 [16:30:59] (03approved) 10raymond-ndibe: [builds-cli] bug fix [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/79 [16:31:02] (03merge) 10raymond-ndibe: [builds-cli] bug fix [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/79 [16:33:56] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [16:37:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [16:41:33] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370465#10003578 (10BTullis)... [16:41:39] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370465#10003579 (10BTullis) [16:51:45] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [16:53:47] !log dcaro@urcuchillay tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud [16:53:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:54:29] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-services-06 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:54:58] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-services-06 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:55:55] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud [16:55:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:00:10] !log andrew@cloudcumin1001 deployment-prep START - Cookbook wmcs.openstack.rebuild_dbinstance [17:02:08] !log andrew@cloudcumin1001 deployment-prep END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) [17:03:36] !log dcaro@urcuchillay tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [17:03:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:03:58] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [17:04:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:05:10] !log andrew@cloudcumin1001 deployment-prep START - Cookbook wmcs.openstack.rebuild_dbinstance [17:05:53] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723#10003743 (10Andrew) [17:06:19] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678 (10RobH) 03NEW [17:07:08] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10003764 (10RobH) [17:07:08] !log andrew@cloudcumin1001 deployment-prep END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) [17:08:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: Upgrade Toolforge apt repository (tools-services hosts) to Debian Bullseye or later - https://phabricator.wikimedia.org/T311914#10003765 (10dcaro) VM up and running, and apt pulling correctly: ` dcaro@tools-bastion-12:~$ sudo ap... [17:11:19] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10003777 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye [17:11:35] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [wikireplicas] frequent replag spikes in clouddb hosts - https://phabricator.wikimedia.org/T367778#10003771 (10fnegri) I spent some more time analyzing long queries running on clouddb1019. I found a few long queries for user `s52168` (`tools.kmlexport... [17:14:12] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10003815 (10cmooney) >>! In T364870#10001348, @ayounsi wrote: > There is an outstanding diff on the switch for `cloudcephmon1006`. It looks correct, but c... [17:21:56] 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: Upgrade Toolforge apt repository (tools-services hosts) to Debian Bullseye or later - https://phabricator.wikimedia.org/T311914#10003853 (10dcaro) 05In progress→03Resolved [17:26:42] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [wikireplicas] frequent replag spikes in clouddb hosts - https://phabricator.wikimedia.org/T367778#10003886 (10fnegri) As an additional test, I killed all the remaining long queries for user `s52168` (`tools.kmlexport`), and 2 long queries for user `u... [17:33:05] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10003913 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye e... [17:34:15] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10003910 (10cmooney) @Jclark-ctr I fixed up some issues in Netbox for cloudcephmon1006 (was on the wrong primary vlan, and had an IP from a cloud-private-... [17:38:50] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3 [17:39:21] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [17:39:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:39:33] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [17:39:35] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3 [17:39:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:43:06] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "dumps" project Buster deprecation - https://phabricator.wikimedia.org/T367528#10003955 (10Andrew) shutting these down due to no response [17:45:11] 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations, 06Release-Engineering-Team: Cloud VPS "integration" project Buster deprecation - https://phabricator.wikimedia.org/T367534#10003968 (10Andrew) 05Open→03Resolved [17:45:26] 06cloud-services-team, 10Toolforge (Toolforge iteration 13): Upgrade Toolforge apt repository (tools-services hosts) to Debian Bullseye or later - https://phabricator.wikimedia.org/T311914#10003970 (10dcaro) I had to restore the packages from the snapshot for some reason, now all of them are there on the r... [17:46:30] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "maps-experiments" project Buster deprecation - https://phabricator.wikimedia.org/T367539#10003980 (10Andrew) 05Open→03Resolved a:03Andrew [17:47:57] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mediawiki-vagrant" project Buster deprecation - https://phabricator.wikimedia.org/T367541#10003989 (10Andrew) I shut down the VM, please delete when you're ready (or I'll delete it in a couple of weeks) [17:48:51] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mwv-apt" project Buster deprecation - https://phabricator.wikimedia.org/T367542#10003991 (10Andrew) Can this project now be deleted? [17:49:59] 10Cloud-VPS (Debian Buster Deprecation), 06Community-Tech, 10IA Upload, 10Wikimedia OCR: Cloud VPS "wikisource" project Buster deprecation - https://phabricator.wikimedia.org/T367564#10004011 (10Andrew) Please delete :) [17:50:45] 10Cloud-VPS (Debian Buster Deprecation), 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Upgrade centralnotice-staging to get off debian buster - https://phabricator.wikimedia.org/T360949#10004015 (10Andrew) I shut down this VM due to lack of response here [17:50:59] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10video2commons: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711#10004016 (10Andrew) *bump* [17:51:28] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Traffic: Replace or remove Debian Buster VMs in 'traffic' cloud-vps project - https://phabricator.wikimedia.org/T360710#10004020 (10Andrew) Mind if I delete traffic-puppetmaster-buster.traffic.eqiad1.wikimedia.cloud? [17:58:41] RESOLVED: CloudVPSDesignateLeaks: Detected 14 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:02:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [18:05:38] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mwv-apt" project Buster deprecation - https://phabricator.wikimedia.org/T367542#10004086 (10Tgr) Yes, thanks. [18:09:59] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [18:10:01] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [18:10:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:10:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:10:18] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [18:17:14] 10wikitech.wikimedia.org: Remove "renameuser" permission from the "bureaucrat" group on Wikitech - https://phabricator.wikimedia.org/T370638#10004125 (10bd808) 05Open→03Declined The Developer account renaming is not exactly the same thing as Wikitech account renaming. There are entanglements however. I d... [18:18:09] (03update) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/16 (owner: 10l10n-bot) [18:18:26] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mwv-apt" project Buster deprecation - https://phabricator.wikimedia.org/T367542#10004130 (10Andrew) 05Open→03Resolved [18:18:40] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10video2commons: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711#10004134 (10Don-vip) Hi Andrew! Sorry for the delay. Thanks a lot for the maintenance of the nfs server :) For the rest... [18:18:56] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [18:19:06] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/16 (owner: 10l10n-bot) [18:19:11] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/16 (owner: 10l10n-bot) [18:25:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:32:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-sgebastion-10 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:45:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:52:05] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [19:00:14] 10Tool-spacemedia, 10Toolforge (Toolforge iteration 13), 10video2commons: Enable SonarCloud usage for GitHub Toolforge projects - https://phabricator.wikimedia.org/T369267#10004352 (10Don-vip) >>! In T369267#10002632, @dcaro wrote: > If you give me your sonarcloud.io user I can add you manually Sure, it's do... [19:16:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:18:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:21:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:54:27] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "dumps" project Buster deprecation - https://phabricator.wikimedia.org/T367528#10004607 (10Nemo_bis) We certainly don't want to be in the way. Feel free to delete the VMs. I was hoping to double check there's nothing to salvage in the local mounts but usually th... [20:36:49] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [20:36:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:37:01] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [20:37:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:07:28] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [21:07:30] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [21:07:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:07:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:16:31] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10005031 (10bd808) >>! In T161859#10002719, @Aklapper wrote: > @Komla: Could you provide some additional info (as you created that wiki page)? Who's the main assignee? TIA It's an e... [21:19:01] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10005035 (10Peachey88) is there a subtask about the emails being sent?, as I saw someone report this on IRC `lang=irc <> I just got an email titled "[IMPORTANT] Action Required for... [21:21:02] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10005047 (10bd808) [21:26:56] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10005069 (10bd808) >>! In T161859#10005035, @Peachey88 wrote: > is there a subtask about the emails being sent?, as I saw someone report this on IRC > > `lang=irc > <> I just got an... [21:49:19] 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations, 10Puppet CI: Cloud VPS "puppet-diffs" project Buster deprecation - https://phabricator.wikimedia.org/T367547#10005121 (10Scott_French) Following up here after various chats on IRC: I'm seeing some unexpected PCC failures for hosts with r... [21:52:45] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10005144 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye [22:01:55] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10005163 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephmon1005.eqiad.wmnet with OS bullseye [22:38:51] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10005224 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye ex... [22:47:54] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10005229 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudcephmon1005.eqiad.wmnet with OS bullseye ex... [23:18:56] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:32:23] 10Cloud-VPS (Debian Buster Deprecation), 06Community-Tech, 10IA Upload, 10Wikimedia OCR: Cloud VPS "wikisource" project Buster deprecation - https://phabricator.wikimedia.org/T367564#10005306 (10Samwilson) 05Open→03Resolved a:03Samwilson Deleted! Sorry for the delay. [23:33:43] 10Cloud-VPS (Debian Buster Deprecation), 10IA Upload, 10Community-Tech (Darwin's Fox (July 15-26, 2024)): Upgrade IA Upload VPS to Bookworm - https://phabricator.wikimedia.org/T369881#10005303 (10Samwilson) 05Open→03Resolved Deleted `ia-upload-prod`. [23:33:51] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [23:33:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:34:03] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [23:34:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:52:57] 10Tool-extjsonuploader: Validate Composer names against Packagist - https://phabricator.wikimedia.org/T370729 (10Samwilson) 03NEW [23:56:13] (03open) 10bd808: gitreview: update section name [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/6 [23:56:38] (03open) 10bd808: gitreview: update section name [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/5 [23:56:49] (03update) 10bd808: gitreview: update section name [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/5 [23:57:36] (03update) 10bd808: gitreview: update section name [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/5 [23:59:17] (03merge) 10jjmc89: gitreview: update section name [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/6 (owner: 10bd808) [23:59:32] (03merge) 10bd808: gitreview: update section name [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/5