[00:23:55] FIRING: MaxConntrack: Max conntrack at 80.94% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:33:55] RESOLVED: MaxConntrack: Max conntrack at 81.41% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:42:55] FIRING: MaxConntrack: Max conntrack at 82.1% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:47:55] RESOLVED: MaxConntrack: Max conntrack at 82.19% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:49:55] FIRING: MaxConntrack: Max conntrack at 81.2% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:54:10] RESOLVED: MaxConntrack: Max conntrack at 80.64% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [01:18:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:33:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:38:44] 10Tool-schedule-deployment: Display link to wikitext diff when reporting a successful patch addition - https://phabricator.wikimedia.org/T367948#10021306 (10bd808) 05Open→03In progress a:03bd808 [01:48:47] (03open) 10bd808: Link to diff after scheduling patch [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/10 (https://phabricator.wikimedia.org/T367948) [01:53:21] (03merge) 10bd808: Link to diff after scheduling patch [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/10 (https://phabricator.wikimedia.org/T367948) [01:58:37] 10Tool-schedule-deployment, 13Patch-For-Review: Display link to wikitext diff when reporting a successful patch addition - https://phabricator.wikimedia.org/T367948#10021315 (10bd808) 05In progress→03Resolved [02:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:18:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:28:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:18:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:33:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:18:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:28:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:33:47] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [wikireplicas] frequent replag spikes in clouddb hosts - https://phabricator.wikimedia.org/T367778#10021481 (10Marostegui) Keep in mind that there are also very big schema changes running on revision table, which is going to create 2-3 days lag for s4. [07:33:27] (03update) 10dsavuljesku: Draft: Bugfixing [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/2 [07:33:53] (03update) 10dsavuljesku: Bugfixing [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/2 [07:34:11] (03merge) 10dcaro: toolforge_get_version: add back wrapper [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/175 [07:35:08] (03merge) 10dcaro: cli: add `jobs-cli` to the user agent [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/56 [07:35:30] (03update) 10dcaro: cli: add host and namespaces to the user agent [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/80 [07:37:10] (03merge) 10dcaro: cli: add host and namespaces to the user agent [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/80 [07:37:28] (03update) 10dcaro: toolforge_get_versions: show component names [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/453 [07:50:58] (03update) 10dcaro: toolforge_get_versions: show component names [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/453 [07:51:02] (03approved) 10dcaro: toolforge_get_versions: show component names [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/453 [07:51:05] (03merge) 10dcaro: toolforge_get_versions: show component names [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/453 [07:51:19] (03open) 10dcaro: version use markdown [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/454 [07:51:28] (03update) 10dsavuljesku: Bugfixing [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/2 [07:54:18] (03merge) 10rvogel: Bugfixing [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/2 (owner: 10dsavuljesku) [08:05:23] (03approved) 10sstefanova: version use markdown [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/454 (owner: 10dcaro) [08:05:25] (03update) 10sstefanova: version use markdown [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/454 (owner: 10dcaro) [08:08:35] (03open) 10sstefanova: components: add ingress-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/178 (https://phabricator.wikimedia.org/T370774) [08:08:43] (03update) 10sstefanova: components: add ingress-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/178 (https://phabricator.wikimedia.org/T370774) [08:09:17] 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: [lima-kilo] add ingress-admission - https://phabricator.wikimedia.org/T370774#10021670 (10Slst2020) 05Open→03In progress [08:16:59] (03update) 10sstefanova: backends: add components-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/28 (https://phabricator.wikimedia.org/T362069) [08:18:10] (03update) 10sstefanova: components: add ingress-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/178 (https://phabricator.wikimedia.org/T370774) [08:18:58] (03update) 10sstefanova: backends: add components-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/28 (https://phabricator.wikimedia.org/T362069) [08:20:25] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 [08:20:34] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 [08:30:14] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#10021740 (10aborrero) Also: * when planing a MR, write results to the gitlab MR as a comment [08:30:30] (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037) [08:31:18] (03update) 10sstefanova: components-api: bump to 0.0.3-20240725133653-f188d2d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/450 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:36:54] (03update) 10sstefanova: backends: add components-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/28 (https://phabricator.wikimedia.org/T362069) [08:42:21] (03update) 10sstefanova: backends: add components-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/28 (https://phabricator.wikimedia.org/T362069) [08:54:19] (03update) 10sstefanova: backends: add components-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/28 (https://phabricator.wikimedia.org/T362069) [09:05:18] (03PS1) 10Stevemunene: Add airflow-analytics-test secret [labs/private] - 10https://gerrit.wikimedia.org/r/1057820 (https://phabricator.wikimedia.org/T371209) [09:19:55] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [09:20:23] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [09:50:15] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#10022126 (10fnegri) > when planing a MR, write results to the gitlab MR as a comment I would rather do it from the CI itself ({T370652}), but I have no objections if y... [09:50:55] (03open) 10dcaro: d/changelog: bump to 0.0.18 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/83 (https://phabricator.wikimedia.org/T370393) [09:53:56] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [09:53:58] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli [09:53:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:53:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:54:46] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [09:54:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:54:49] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli [09:54:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:55:52] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [09:55:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:56:10] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [09:56:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:56:49] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [09:56:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:56:53] 10VPS-Projects: Exception trying to upload file to Commons Archive at commonsarchive.wmflabs.org - https://phabricator.wikimedia.org/T280807#10022149 (10Bugreporter) 05Stalled→03Declined Project no longer exists per {T306064}. [09:57:03] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli [09:57:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:58:51] (03close) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037) [09:59:37] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [09:59:37] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [09:59:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:59:47] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli [09:59:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:00:10] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:02:08] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [10:02:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:02:20] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli [10:02:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:02:49] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [10:02:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:02:59] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [10:03:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:06:38] (03update) 10sstefanova: calico: upgrade to 3.26.4 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/8 (https://phabricator.wikimedia.org/T370046) [10:18:44] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [10:18:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:19:01] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli [10:19:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:21:09] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [10:21:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:21:48] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [10:21:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:24:24] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [10:24:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:25:04] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli [10:25:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:59:24] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [10:59:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:00:21] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [11:00:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:02:19] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [11:02:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:03:11] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [11:03:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:04:15] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [11:04:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:05:10] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [11:05:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:07:51] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [11:07:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:08:46] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [11:08:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:10:04] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [11:10:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:11:06] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [11:11:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:12:55] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [11:12:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:13:51] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [11:13:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:17:15] (03PS1) 10David Caro: toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 [11:20:19] (03CR) 10CI reject: [V:04-1] toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 (owner: 10David Caro) [11:23:44] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [11:24:54] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [11:25:42] (03PS2) 10David Caro: toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 [11:25:47] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [11:25:49] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli [11:25:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:25:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:26:04] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [11:26:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:26:13] (03PS3) 10David Caro: toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 [11:27:06] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [11:27:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:28:28] (03PS4) 10David Caro: toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 [11:31:15] (03CR) 10CI reject: [V:04-1] toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 (owner: 10David Caro) [11:34:20] 06cloud-services-team: openstack: codfw1dev: nova-compute can't contact rabbitmq - https://phabricator.wikimedia.org/T371242 (10aborrero) 03NEW [11:34:42] (03PS5) 10David Caro: toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 [11:36:01] 06cloud-services-team: openstack: codfw1dev: nova-compute can't contact rabbitmq - https://phabricator.wikimedia.org/T371242#10022549 (10aborrero) Error on nova-scheduler: ` Jul 29 11:29:10 cloudcontrol2005-dev nova-scheduler[2262069]: 2024-07-29 11:29:10.008 2262069 ERROR oslo.messaging._drivers.impl_rabbit [-... [11:36:42] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [11:38:04] (03CR) 10CI reject: [V:04-1] toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 (owner: 10David Caro) [11:38:32] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [11:41:35] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [11:41:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:42:36] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [11:42:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:43:58] (03merge) 10dcaro: version use markdown [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/454 [11:50:39] (03PS6) 10David Caro: toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 [11:53:54] (03CR) 10CI reject: [V:04-1] toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 (owner: 10David Caro) [11:58:19] 10Cloud-VPS: Adding a member to a project takes ~ 40 seconds - https://phabricator.wikimedia.org/T371245 (10hashar) 03NEW [11:59:38] (03PS7) 10David Caro: toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 [12:00:29] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [12:00:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:01:25] !log dcaro@urcuchillay tools END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli [12:01:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:02:04] 10Cloud-VPS: Adding a member to a project takes ~ 40 seconds - https://phabricator.wikimedia.org/T371245#10022609 (10hashar) 05Open→03Declined That must have been transient since the last few ones took less than 10 seconds ;) [12:02:58] (03CR) 10CI reject: [V:04-1] toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 (owner: 10David Caro) [12:07:08] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli [12:07:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:08:13] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli [12:08:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:12:56] (03PS8) 10David Caro: toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 [12:13:28] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#10022661 (10Jgiannelos) [12:14:18] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Rebuild or delete deployment-docker-mobileapps01 - https://phabricator.wikimedia.org/T369915#10022657 (10Jgiannelos) 05Open→03Resolved a:03Jgiannelos Yeah i just deleted it. [12:17:29] (03update) 10sstefanova: components-api: bump to 0.0.3-20240725133653-f188d2d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/450 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:20:24] (03open) 10raymond-ndibe: [toolforge-weld] add custom resources version to k8sclient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/51 (https://phabricator.wikimedia.org/T359650) [12:21:36] (03update) 10raymond-ndibe: [toolforge-weld] add custom resources version to k8sclient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/51 (https://phabricator.wikimedia.org/T359650) [12:25:30] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/18 [12:28:58] (03approved) 10dcaro: d/changelog: bump to 0.0.18 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/83 (https://phabricator.wikimedia.org/T370393) [12:29:08] (03merge) 10dcaro: d/changelog: bump to 0.0.18 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/83 (https://phabricator.wikimedia.org/T370393) [12:29:17] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10022695 (10Jgiannelos) Hi, i think this was an attempt to quickly ass... [12:38:30] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli [12:38:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:38:34] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli [12:38:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:39:09] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli [12:39:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:39:13] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli [12:39:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:41:26] (03open) 10dcaro: d/changelog: bump to 16.0.16 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/57 [12:41:51] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli [12:41:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:41:55] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli [12:41:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:42:06] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli [12:42:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:43:08] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli [12:43:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:46:45] 06cloud-services-team: openstack: codfw1dev: nova-compute can't contact rabbitmq - https://phabricator.wikimedia.org/T371242#10022765 (10aborrero) For completeness, all hypervisors are marked as down: ` root@cloudcontrol2005-dev:/var/log/rabbitmq# wmcs-openstack hypervisor list +--------------------------------... [12:48:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:05:56] FIRING: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:10:56] RESOLVED: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:18:14] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli [13:18:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:19:23] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli [13:19:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:32:43] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: [wmcs-backup] Backup snapshots of deleted volumes are never cleaned up - https://phabricator.wikimedia.org/T358774#10022914 (10fnegri) a:05fnegri→03None [13:33:04] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: [wmcs-backup] Race condition between backup and cleanup timers - https://phabricator.wikimedia.org/T358780#10022916 (10fnegri) a:05fnegri→03None [13:44:11] 06cloud-services-team: openstack: codfw1dev: nova-compute can't contact rabbitmq - https://phabricator.wikimedia.org/T371242#10022943 (10Andrew) Can you tell me what (if any) change or update preceded this? [14:03:24] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.openstack.rebuild_dbinstance [14:05:21] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) [14:10:53] (03approved) 10dcaro: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 (owner: 10raymond-ndibe) [14:10:58] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723#10023135 (10Andrew) [14:10:59] (03update) 10dcaro: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 (owner: 10raymond-ndibe) [14:11:39] (03approved) 10dcaro: d/changelog: bump to 16.0.16 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/57 [14:11:42] (03update) 10dcaro: d/changelog: bump to 16.0.16 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/57 [14:11:42] (03merge) 10dcaro: d/changelog: bump to 16.0.16 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/57 [14:12:48] (03update) 10dcaro: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 (owner: 10raymond-ndibe) [14:13:14] (03update) 10dcaro: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) (owner: 10raymond-ndibe) [14:13:40] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723#10023131 (10Andrew) 05Open→03Resolved Everything is now based off of trove-guest-bobcat-ubuntu-jammy. That image contains a local application of pro... [14:13:43] (03update) 10dcaro: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) (owner: 10raymond-ndibe) [14:15:15] (03update) 10raymond-ndibe: [toolforge-weld] support back python 3.7 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/50 (https://phabricator.wikimedia.org/T370932) [14:17:14] (03update) 10raymond-ndibe: [toolforge-weld] support back python 3.7 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/50 (https://phabricator.wikimedia.org/T370932) [14:20:32] 06cloud-services-team: openstack: codfw1dev: nova-compute can't contact rabbitmq - https://phabricator.wikimedia.org/T371242#10023184 (10aborrero) >>! In T371242#10022943, @Andrew wrote: > Can you tell me what (if any) change or update preceded this? I updated python3-nova on all servers: {T371240} [14:23:39] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2024-07-23 - https://phabricator.wikimedia.org/T370760#10023189 (10fnegri) 05In progress→03Resolved Replication resumed at around 2024-07-27 23:30 UTC and caught up with the primary around 16 hou... [14:25:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:26:11] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [14:26:54] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:27:31] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [14:28:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:29:34] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [14:29:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:31:20] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [14:33:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:34:14] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [14:37:25] (03merge) 10dcaro: builds: override default imagemagick config [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/56 (https://phabricator.wikimedia.org/T370610) [14:38:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:39:16] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [14:39:17] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.114-20240729143738-4f948212 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/455 (https://phabricator.wikimedia.org/T370610) [14:42:43] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [14:42:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:42:58] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder [14:43:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:45:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [14:49:23] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [14:54:03] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Cumin, 06Infrastructure-Foundations, 13Patch-For-Review: [cumin] [openstack] Openstack backend fails when project is not set - https://phabricator.wikimedia.org/T346453#10023426 (10joanna_borun) p:05High→03Medium [15:03:08] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [15:03:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:03:26] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder [15:03:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:09:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [15:12:19] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [15:12:55] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [15:12:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:13:09] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder [15:13:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:15:17] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [15:15:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:15:27] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder [15:15:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:16:24] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [15:16:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:16:42] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [15:16:43] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder [15:16:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:18:30] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [15:19:34] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [15:22:42] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [15:23:15] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [15:25:23] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [15:25:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:25:46] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder [15:25:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:33:12] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [15:33:50] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [15:40:31] 06cloud-services-team: openstack: codfw1dev: manually clean up old hypervisors - https://phabricator.wikimedia.org/T371274 (10aborrero) 03NEW [15:44:52] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [15:45:34] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [15:59:17] (03open) 10dcaro: deploy: allow running from any directory [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/456 [16:02:46] (03update) 10dcaro: builds-builder: bump to 0.0.114-20240729143738-4f948212 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/455 (https://phabricator.wikimedia.org/T370610) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:03:03] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [16:03:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:03:28] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder [16:03:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:05:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [16:07:07] (03update) 10dcaro: builds-builder: bump to 0.0.114-20240729143738-4f948212 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/455 (https://phabricator.wikimedia.org/T370610) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:07:13] (03close) 10dcaro: deploy: allow running from any directory [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/456 [16:07:24] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [16:07:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:07:48] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder [16:07:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:09:35] (03update) 10dcaro: builds-builder: bump to 0.0.114-20240729143738-4f948212 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/455 (https://phabricator.wikimedia.org/T370610) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:09:37] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [16:09:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:10:15] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [16:10:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:10:28] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [16:11:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [16:15:12] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10023776 (10jijiki) 05Open→03In progress [16:15:42] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10023780 (10jijiki) [16:15:51] 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10023781 (10jijiki) [16:16:09] (03PS9) 10David Caro: toolforge.deploy: support deploying packages [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 [16:16:24] (03CR) 10David Caro: "Finished adding recorded tests, ready" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1057847 (owner: 10David Caro) [16:16:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [16:17:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [16:22:36] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [16:24:04] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [16:24:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:24:52] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [16:24:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:32:21] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [16:32:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:32:54] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [16:32:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:33:16] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [16:33:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:33:47] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [16:33:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:38:38] (03approved) 10dcaro: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) (owner: 10raymond-ndibe) [16:41:33] (03approved) 10dcaro: builds-builder: bump to 0.0.114-20240729143738-4f948212 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/455 (https://phabricator.wikimedia.org/T370610) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:41:34] (03update) 10dcaro: builds-builder: bump to 0.0.114-20240729143738-4f948212 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/455 (https://phabricator.wikimedia.org/T370610) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:41:37] (03merge) 10dcaro: builds-builder: bump to 0.0.114-20240729143738-4f948212 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/455 (https://phabricator.wikimedia.org/T370610) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:44:03] 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10023964 (10dcaro) p:05Triage→03Medium [16:44:37] 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10023973 (10dcaro) This is deployed now, now the memory limit used is the one of the running worker node (and as high a... [16:46:33] 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10023963 (10dcaro) [16:48:11] 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10023967 (10dcaro) 05Open→03Resolved a:03dcaro [16:48:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:54:26] (03approved) 10dcaro: [toolforge-weld] support back python 3.7 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/50 (https://phabricator.wikimedia.org/T370932) (owner: 10raymond-ndibe) [16:54:28] (03update) 10dcaro: [toolforge-weld] support back python 3.7 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/50 (https://phabricator.wikimedia.org/T370932) (owner: 10raymond-ndibe) [17:07:06] (03open) 10dcaro: toolforge_get_versions: add component bang to the mr number [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/457 [17:13:07] 10VPS-project-Codesearch: Codesearch excluded files link broken - https://phabricator.wikimedia.org/T371288 (10Pppery) 03NEW [17:13:12] (03approved) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 [17:13:15] (03merge) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 [17:13:17] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [17:14:26] (03update) 10dcaro: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) (owner: 10raymond-ndibe) [17:16:14] (03update) 10dcaro: toolforge_deploy_mr: add toolforge-weld package [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/176 [17:16:43] (03approved) 10dcaro: components: add ingress-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/178 (https://phabricator.wikimedia.org/T370774) (owner: 10sstefanova) [17:17:22] (03merge) 10dcaro: toolforge_deploy_mr: add toolforge-weld package [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/176 [17:20:25] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [17:22:24] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [17:22:27] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [17:22:46] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [17:24:14] (03approved) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [17:24:19] (03merge) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [17:24:20] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [17:24:51] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [17:26:13] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [17:26:22] (03update) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 [17:26:32] (03update) 10raymond-ndibe: [lima-kilo] update bookworm arm64 image [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/155 [17:27:16] (03update) 10raymond-ndibe: [toolforge-weld] support back python 3.7 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/50 (https://phabricator.wikimedia.org/T370932) [17:27:23] (03approved) 10raymond-ndibe: [toolforge-weld] support back python 3.7 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/50 (https://phabricator.wikimedia.org/T370932) [17:27:32] (03merge) 10raymond-ndibe: [toolforge-weld] support back python 3.7 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/50 (https://phabricator.wikimedia.org/T370932) [17:28:08] (03approved) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) [17:28:13] (03update) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) [17:28:21] (03merge) 10raymond-ndibe: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) [17:28:24] (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [17:30:10] 10Striker: Rotate StrikerBot GitLab PAT before it expires on 2024-08-01 - https://phabricator.wikimedia.org/T371289 (10bd808) 03NEW [17:30:44] 10Striker: Rotate StrikerBot GitLab PAT before it expires on 2024-08-01 - https://phabricator.wikimedia.org/T371289#10024237 (10bd808) [17:31:51] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.324-20240729172834-18574769 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/458 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) [17:32:04] (03update) 10sstefanova: components: add ingress-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/178 (https://phabricator.wikimedia.org/T370774) [17:33:22] (03merge) 10sstefanova: components: add ingress-admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/178 (https://phabricator.wikimedia.org/T370774) [17:34:52] 10Striker: Rotate StrikerBot GitLab PAT before it expires on 2024-08-01 - https://phabricator.wikimedia.org/T371289#10024249 (10bd808) p:05Triage→03High This token is used by #striker to enable management of GitLab projects. It will expire annually going forward due to GitLab's product decision to only allow... [17:52:50] (03open) 10raymond-ndibe: Tag v1.6.0 release [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/52 [18:01:53] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [18:01:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:02:42] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [18:02:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:05:46] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [18:05:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:06:36] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [18:06:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:12:23] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.324-20240729172834-18574769 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/458 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [18:12:34] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.324-20240729172834-18574769 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/458 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [18:13:41] (03approved) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [18:13:46] (03merge) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) [18:16:06] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.325-20240729181356-e2f35e86 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/459 (https://phabricator.wikimedia.org/T366209) [18:21:40] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [18:21:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:22:28] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [18:22:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:23:32] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [18:23:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:24:21] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [18:24:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:31:30] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.325-20240729181356-e2f35e86 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/459 (https://phabricator.wikimedia.org/T366209) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [18:31:34] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.325-20240729181356-e2f35e86 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/459 (https://phabricator.wikimedia.org/T366209) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [18:31:39] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.325-20240729181356-e2f35e86 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/459 (https://phabricator.wikimedia.org/T366209) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [18:31:57] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.324-20240729172834-18574769 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/458 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [18:32:17] (03update) 10raymond-ndibe: toolforge_get_versions: add component bang to the mr number [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/457 (owner: 10dcaro) [18:44:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [18:47:36] (03approved) 10raymond-ndibe: toolforge_get_versions: add component bang to the mr number [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/457 (owner: 10dcaro) [18:47:41] (03update) 10raymond-ndibe: toolforge_get_versions: add component bang to the mr number [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/457 (owner: 10dcaro) [18:51:06] 10Cloud-VPS (Debian Buster Deprecation), 10Google-api-proxy, 10Community-Tech (Darwin's Fox (July 15-26, 2024)): Cloud VPS "google-api-proxy" project Buster deprecation - https://phabricator.wikimedia.org/T367532#10024749 (10MusikAnimal) We are trying to find someone with the expertise to assist us in creati... [18:52:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [18:55:08] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [19:24:50] 10Cloud-VPS (Debian Buster Deprecation), 10Google-api-proxy, 10Community-Tech (Darwin's Fox (July 15-26, 2024)): Cloud VPS "google-api-proxy" project Buster deprecation - https://phabricator.wikimedia.org/T367532#10024967 (10bd808) >>! In T367532#10024748, @MusikAnimal wrote: > We are trying to find someone... [19:31:46] 10Cloud-VPS (Debian Buster Deprecation), 10Google-api-proxy, 10Community-Tech (Darwin's Fox (July 15-26, 2024)): Cloud VPS "google-api-proxy" project Buster deprecation - https://phabricator.wikimedia.org/T367532#10025003 (10MusikAnimal) a:03MusikAnimal Ah! How quickly I forget :) Thank you @bd808! I'm goi... [19:44:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:44:18] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:49:18] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:02:47] 10Striker: Rotate StrikerBot GitLab PAT before it expires on 2024-08-01 - https://phabricator.wikimedia.org/T371289#10025174 (10bd808) I have provisioned a new Personal Access Token (PAT) for the https://gitlab.wikimedia.org/striker user named "Striker API 2024". It will expire on 2025-07-28. The PAT value is av... [20:06:08] 06cloud-services-team, 10Striker: Rotate StrikerBot GitLab PAT before it expires on 2024-08-01 - https://phabricator.wikimedia.org/T371289#10025188 (10bd808) [20:48:56] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:51:23] 06cloud-services-team, 10Striker: Rotate StrikerBot GitLab PAT before it expires on 2024-08-01 - https://phabricator.wikimedia.org/T371289#10025421 (10Andrew) 05Open→03Resolved a:03Andrew done: ` commit 44e084812e65e274c729da8a6b16ab5beb9782b0 (HEAD -> master) Author: gitpuppet for private repo 06cloud-services-team: openstack: codfw1dev: manually clean up old hypervisors - https://phabricator.wikimedia.org/T371274#10025440 (10Andrew) 05Open→03Resolved a:03Andrew I removed those two old servers via the database. I don't know why the CLI doesn't work (the cloudvirts were empty). [22:09:11] 10Cloud-VPS (Quota-requests), 06collaboration-services: Increase quota for volumes in devtools project - https://phabricator.wikimedia.org/T371093#10025637 (10Dzahn) [22:16:17] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Update Wikitech's LDAP credentials to be read-only - https://phabricator.wikimedia.org/T367287#10025647 (10jijiki) [22:46:09] 10Cloud-VPS (Debian Buster Deprecation), 10Google-api-proxy, 10Community-Tech (Darwin's Fox (July 15-26, 2024)): Cloud VPS "google-api-proxy" project Buster deprecation - https://phabricator.wikimedia.org/T367532#10025693 (10MusikAnimal) 05Open→03Resolved Declaring this resolved! No QA necessary. Tha... [23:19:41] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:38:33] (03CR) 10Krinkle: "The qqq.json file documents how the message should be translated in the future. This currently mentions strftime still. Can you update thi" [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1055408 (https://phabricator.wikimedia.org/T331468) (owner: 10GergesShamon)