[00:02:31] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T370114#9983725 (10LibUp-bot) [00:02:33] 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T370115#9983727 (10LibUp-bot) [01:30:43] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [01:32:33] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29688 bytes in 0.662 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [01:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:04:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:03:12] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9983905 (10Andrew) [03:13:28] FIRING: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:06:56] FIRING: SystemdUnitDown: The service unit purge_vm_rbd_images.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:13:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:15:36] (03close) 10legoktm: Remove duplicate healthz route [toolforge-repos/logo-test] - 10https://gitlab.wikimedia.org/toolforge-repos/logo-test/-/merge_requests/2 (https://phabricator.wikimedia.org/T363216) (owner: 10taavi) [04:16:39] 10Tool-logo-test: logo-test is down - https://phabricator.wikimedia.org/T363216#9983915 (10Legoktm) 05Open→03Resolved a:03Legoktm Fixed in https://gitlab.wikimedia.org/toolforge-repos/logo-test/-/commit/210df3f7cff6173b1d2e22505a4a044bec8defc3 (sorry I didn't see this task until just now). [05:10:28] FIRING: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:15:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:01:56] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:10:06] 10Cloud-VPS (Debian Buster Deprecation), 10IA Upload, 10Community-Tech (Darwin's Fox (July 15-26, 2024)): Upgrade IA Upload VPS to Bookworm - https://phabricator.wikimedia.org/T369881#9983992 (10Samwilson) a:03Samwilson [07:19:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:21:15] (03PS3) 10David Caro: depool_and_destroy: also zap the devices [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054376 [08:22:14] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [08:22:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:25:03] (03update) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 [08:27:14] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [08:27:16] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [08:27:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:27:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:29:14] (03open) 10dcaro: ci: avoid variable interpolation when selecting runner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/45 [08:30:36] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [08:30:38] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [08:30:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:30:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:33:34] (03approved) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 [08:33:37] (03merge) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 [08:33:38] (03update) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 [08:35:47] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [08:35:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:36:06] (03approved) 10aborrero: ci: avoid variable interpolation when selecting runner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/45 (owner: 10dcaro) [08:36:25] 10Cloud-VPS (Debian Buster Deprecation), 10IA Upload, 10Community-Tech (Darwin's Fox (July 15-26, 2024)): Upgrade IA Upload VPS to Bookworm - https://phabricator.wikimedia.org/T369881#9984229 (10Samwilson) I've migrated the tool to two new servers, `ia-upload-prod2` and `ia-upload-test2` and updated the web... [08:37:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [08:37:16] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [08:37:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:39:08] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.163-20240716083349-0dc2fa69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/418 (https://phabricator.wikimedia.org/T367181) [08:39:12] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.163-20240716083349-0dc2fa69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/418 (https://phabricator.wikimedia.org/T367181) [08:42:18] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [08:42:20] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [08:42:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:42:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:44:36] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [08:44:38] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [08:44:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:44:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:46:59] (03PS1) 10David Caro: bootstrap_and_add: skip host if no new devices found [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054494 [08:47:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [08:48:11] 10Cloud-VPS (Debian Buster Deprecation), 06Machine-Learning-Team: Cloud VPS "machine-learning" project Buster deprecation - https://phabricator.wikimedia.org/T367537#9984273 (10klausman) 05Open→03Resolved [08:48:40] (03merge) 10dcaro: ci: avoid variable interpolation when selecting runner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/45 [08:48:42] (03update) 10dcaro: ci: avoid variable interpolation when selecting runner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/45 [08:48:58] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-admission: bump to 0.0.14-20240716084546-0b645f15 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/419 (https://phabricator.wikimedia.org/T369890) [08:49:39] (03open) 10aborrero: gitlab-ci.yml: select runner with the RUNNER_TAG var [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/9 [08:50:24] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [08:50:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:50:44] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission [08:50:53] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission [08:51:08] (03approved) 10dcaro: gitlab-ci.yml: select runner with the RUNNER_TAG var [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/9 (owner: 10aborrero) [08:51:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [08:51:11] (03update) 10dcaro: gitlab-ci.yml: select runner with the RUNNER_TAG var [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/9 (owner: 10aborrero) [08:52:22] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission [08:52:33] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission [08:53:57] 10Cloud-VPS (Quota-requests): Request new flavor for integration project - https://phabricator.wikimedia.org/T370127 (10hashar) 03NEW [08:54:18] 10Cloud-VPS (Quota-requests): Request new flavor for integration project - https://phabricator.wikimedia.org/T370127#9984306 (10hashar) [08:55:13] (03merge) 10aborrero: envvars-admission: bump to 0.0.14-20240716084546-0b645f15 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/419 (https://phabricator.wikimedia.org/T369890) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:56:31] (03merge) 10aborrero: gitlab-ci.yml: select runner with the RUNNER_TAG var [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/9 [08:59:45] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-admission: bump to 0.0.15-20240716085643-ebe65602 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/420 [09:01:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: toolforge: kubernetes fails to handle some pods that are being mutated by our admission controllers - https://phabricator.wikimedia.org/T369890#9984330 (10aborrero) 05In progress→03Resolved a:03aborrero [09:06:47] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16 [09:07:11] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16 [09:10:04] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16 [09:10:34] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [09:10:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:12:10] (03open) 10dcaro: ci: remove unused variable MEMORY_OPTIMIZED [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/104 [09:14:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:15:28] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [09:15:30] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [09:15:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:15:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:17:17] 10Cloud-VPS, 06Infrastructure-Foundations, 10Puppet-Core: Puppet self role on integration WMCS project fails to match certificates - https://phabricator.wikimedia.org/T370130 (10hashar) 03NEW [09:17:40] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16 [09:17:47] 10Cloud-VPS, 06Infrastructure-Foundations, 10Puppet-Core: Puppet self role on integration WMCS project fails to match certificates - https://phabricator.wikimedia.org/T370130#9984392 (10hashar) [09:18:47] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [09:18:49] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [09:18:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:18:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:19:02] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [09:19:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:19:14] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [09:19:14] RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:19:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:19:19] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) [09:19:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:21:14] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:26:14] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:28:51] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16 [09:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 6 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:31:37] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [09:31:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:32:39] 10Cloud-VPS, 06Infrastructure-Foundations, 10Puppet-Core: Puppet self role on integration WMCS project fails to match certificates - https://phabricator.wikimedia.org/T370130#9984420 (10hashar) 05Open→03Resolved a:03hashar Eventually on `integration-puppetserver-01.integration.eqiad1.wikimedia.clou... [09:32:44] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:36:20] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [09:36:22] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [09:36:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:36:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:36:26] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) [09:36:29] RESOLVED: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:36:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:37:38] 10Cloud-VPS (Quota-requests): Request quota increase for huma project - https://phabricator.wikimedia.org/T370010#9984437 (10Ladsgroup) Thank you! [09:39:35] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16 [09:41:19] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16 [09:45:44] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:46:31] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [09:46:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:46:36] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) [09:46:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:48:54] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16 [09:50:41] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 [09:50:42] !log aborrero@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 [09:50:44] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-control-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:51:04] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 [09:52:11] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 [10:00:35] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.wait_for_rebalance [10:00:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:00:56] (03PS1) 10David Caro: ceph.checks: add extra logs for easy following [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054508 [10:00:57] (03PS1) 10David Caro: bootstrap_and_add: Use correct device path [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054509 [10:00:57] (03PS1) 10David Caro: ceph.wait_for_rebalance: new handy cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054510 [10:03:40] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [10:03:55] (03CR) 10CI reject: [V:04-1] bootstrap_and_add: Use correct device path [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054509 (owner: 10David Caro) [10:04:06] (03CR) 10CI reject: [V:04-1] ceph.wait_for_rebalance: new handy cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054510 (owner: 10David Caro) [10:05:23] 10Cloud-VPS: Horizon does not commit changes to cloud/instance-puppet git repo since June 24th 2024 - https://phabricator.wikimedia.org/T370136 (10hashar) 03NEW [10:07:29] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 [10:08:34] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 [10:08:59] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 [10:09:32] (03PS2) 10David Caro: bootstrap_and_add: Use correct device path [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054509 [10:09:33] (03PS2) 10David Caro: ceph.wait_for_rebalance: new handy cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054510 [10:09:54] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 [10:10:02] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 [10:10:18] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 [10:10:57] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 [10:10:58] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 [10:10:58] !log aborrero@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 [10:10:59] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 [10:10:59] !log aborrero@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 [10:11:24] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 [10:11:27] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 [10:11:40] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 [10:12:33] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 [10:12:34] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 [10:12:37] (03CR) 10CI reject: [V:04-1] ceph.wait_for_rebalance: new handy cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054510 (owner: 10David Caro) [10:12:46] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 [10:13:21] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 [10:13:39] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 [10:13:40] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 [10:13:55] (03update) 10sstefanova: kind: upgrade to k8s 1.25 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/161 (https://phabricator.wikimedia.org/T369165) [10:14:00] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission [10:14:04] (03update) 10sstefanova: kind: upgrade to k8s 1.25 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/161 (https://phabricator.wikimedia.org/T369165) [10:14:10] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission [10:14:14] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [10:14:23] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission [10:14:27] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 [10:14:34] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission [10:14:44] (03merge) 10aborrero: envvars-admission: bump to 0.0.15-20240716085643-ebe65602 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/420 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:14:44] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 [10:14:45] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 [10:15:15] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 [10:16:18] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 [10:16:28] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 [10:17:34] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 [10:18:46] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 [10:19:54] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 [10:20:32] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 [10:20:48] !log aborrero@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 [10:21:24] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0) [10:21:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:21:40] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 [10:22:36] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 [10:23:39] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 [10:23:50] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 [10:24:54] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 [10:25:13] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 [10:25:43] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [10:25:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:26:24] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 [10:26:35] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 [10:27:41] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 [10:28:35] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 [10:29:42] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 [10:29:51] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 [10:31:00] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 [10:31:36] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 [10:32:42] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 [10:32:58] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 [10:34:03] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 [10:34:22] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 [10:34:49] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 [10:35:28] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 [10:35:54] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 [10:35:55] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 [10:35:57] (03open) 10dcaro: ci: force memory-optimized runners [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/41 [10:36:23] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9984656 (10fnegri) [10:36:51] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 [10:36:59] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 [10:37:00] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 [10:37:56] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 [10:38:08] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 [10:38:08] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 [10:38:09] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 [10:38:45] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9984673 (10fnegri) @BTullis I think you can proceed with your test and turn off all sections for a week. When that is done and you are confident nothi... [10:39:11] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 [10:39:15] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 [10:39:15] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 [10:39:26] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 [10:40:22] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 [10:40:23] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 [10:40:32] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 [10:40:55] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 [10:41:28] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 [10:41:29] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 [10:41:59] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 [10:42:09] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 [10:42:10] (03update) 10dcaro: run_functional_tests: when custom tool is passed, set the uid too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/410 [10:42:15] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9984678 (10BTullis) >>! In T365424#9984673, @fnegri wrote: > @BTullis I think you can proceed with your test and turn off all sections for a week. Whe... [10:42:34] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 [10:42:35] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 [10:43:10] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 [10:43:40] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 [10:43:41] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 [10:43:59] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9984679 (10fnegri) I think it makes sense to do your test first. I can change back the role before the reimage. [10:44:17] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 [10:44:45] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 [10:44:46] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 [10:45:23] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 [10:45:51] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 [10:45:52] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 [10:46:08] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 [10:46:37] (03approved) 10sstefanova: run_functional_tests: when custom tool is passed, set the uid too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/410 (owner: 10dcaro) [10:46:39] (03update) 10sstefanova: run_functional_tests: when custom tool is passed, set the uid too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/410 (owner: 10dcaro) [10:46:49] (03merge) 10dcaro: run_functional_tests: when custom tool is passed, set the uid too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/410 [10:46:54] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 [10:46:55] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 [10:47:13] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 [10:47:57] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [10:48:00] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 [10:48:01] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 [10:48:47] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29685 bytes in 0.206 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [10:49:10] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 [10:49:11] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 [10:49:18] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 [10:50:18] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 [10:50:19] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 [10:50:34] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 [10:50:48] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 [10:51:20] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 [10:51:23] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 [10:51:52] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 [10:52:38] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 [10:53:45] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 [10:54:29] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 [10:55:30] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 [10:56:47] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 [10:57:22] !log aborrero@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 [10:57:48] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 [10:57:55] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [10:57:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:58:43] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 [10:59:49] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 [11:00:14] (03open) 10dcaro: metrics: listen on port 9000 too [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/104 [11:02:41] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [11:02:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:02:53] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 [11:02:53] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [11:02:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:05:09] (03PS1) 10Slyngshede: P:idm_test add dummy secrets for mediawiki integration. [labs/private] - 10https://gerrit.wikimedia.org/r/1054519 [11:08:08] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 [11:08:28] FIRING: [2x] InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:09:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [11:10:06] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 [11:10:08] !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 [11:11:13] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 [11:12:03] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 [11:12:27] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 [11:12:43] (03approved) 10sstefanova: metrics: listen on port 9000 too [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/104 (owner: 10dcaro) [11:13:29] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 [11:14:05] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 [11:14:24] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [11:14:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:15:10] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 [11:15:34] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 [11:16:39] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 [11:18:28] FIRING: [2x] InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:19:46] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [11:19:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:20:17] FIRING: HarborComponentDown: No data about Harbor components found. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown [11:20:23] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 [11:20:49] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.wait_for_rebalance [11:20:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:21:32] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 [11:22:00] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 [11:22:56] !log dcaro@urcuchillay tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [11:22:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:23:11] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 [11:23:28] RESOLVED: [2x] InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:23:33] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 [11:23:51] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [11:23:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:24:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [11:24:37] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 [11:25:17] RESOLVED: HarborComponentDown: No data about Harbor components found. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown [11:25:23] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 [11:25:53] !log dcaro@urcuchillay tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [11:25:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:26:31] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 [11:27:28] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 [11:28:33] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 [11:28:53] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 [11:30:00] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 [11:30:14] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 [11:31:23] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 [11:33:55] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [11:33:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:34:31] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [11:34:41] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [11:35:51] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [11:36:02] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [12:43:48] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:19:49] (03open) 10rvogel: Add wrapper script for manual execution [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/1 [13:21:18] (03update) 10rvogel: Add wrapper script for manual execution [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/1 [13:21:45] (03update) 10rvogel: Add wrapper script for manual execution [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/1 [13:23:10] 10Toolforge (Toolforge iteration 12): [toolforge-deploy] envvars functional tests fail when out of quota - https://phabricator.wikimedia.org/T367169#9985173 (10dcaro) 05Declined→03Resolved [13:24:39] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade control plane nodes to k8s 1.25 - https://phabricator.wikimedia.org/T369172#9985183 (10aborrero) 05Open→03Resolved a:03aborrero [13:25:37] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade worker nodes to k8s 1.25 - https://phabricator.wikimedia.org/T369171#9985197 (10aborrero) 05Open→03In progress p:05Triage→03High only ingress nodes are left. [13:40:02] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 12), 05Goal: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9985350 (10dcaro) a:03dcaro [13:41:37] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0) [13:41:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:42:27] 10Tool-spacemedia, 06Commons, 10MediaWiki-File-management: Make image hashes available through API or database query on Commons - https://phabricator.wikimedia.org/T167947#9985347 (10Prototyperspective) This would be very useful and have a large impact in WMC. It could and should also be used by scripts and... [13:46:47] 10Toolforge (Toolforge iteration 12): [toolforge deploy] direct-api tests fail intermittently on toolsbeta - https://phabricator.wikimedia.org/T369891#9985442 (10dcaro) p:05Medium→03Low [13:48:57] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T370114#9985448 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/441 [13:49:09] vivian-rook opened https://github.com/toolforge/paws/pull/441 [13:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:50:45] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: get a working setup for ingress-nginx and webservices in lima-kilo - https://phabricator.wikimedia.org/T369363#9985456 (10dcaro) p:05Triage→03Medium [13:51:57] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: get a working setup for ingress-nginx and webservices in lima-kilo - https://phabricator.wikimedia.org/T369363#9985459 (10dcaro) [13:56:22] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: integrate fourohfour as a custom component, rather than a normal tool - https://phabricator.wikimedia.org/T369364#9985485 (10dcaro) p:05Triage→03Low [13:57:49] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Toolforge, 10Observability-Alerting, 05Goal: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#9985504 (10fgiunchedi) A very low hanging fruit to make progress on this task is the following prometheus-b... [13:57:55] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [13:58:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:02:07] 10Cloud-VPS (Quota-requests): Request new flavor for integration project - https://phabricator.wikimedia.org/T370127#9985522 (10fnegri) +1 [14:02:10] 10Cloud-VPS (Quota-requests): Request new flavor for integration project - https://phabricator.wikimedia.org/T370127#9985525 (10Slst2020) a:03Slst2020 [14:02:11] 10Cloud-VPS (Quota-requests): Request new flavor for integration project - https://phabricator.wikimedia.org/T370127#9985524 (10aborrero) LGTM. +1. [14:08:14] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 [14:08:34] (03open) 10dcaro: fix webservice [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/169 [14:09:09] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 [14:09:19] (03update) 10dcaro: Handle tools-webservice package [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/169 [14:10:11] (03update) 10dcaro: toolforge_deploy_mr: make install/uninstall non-interactive [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/169 [14:10:17] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 [14:11:05] (03update) 10dcaro: toolforge_deploy_mr: make install/uninstall non-interactive [repos/cloud/toolforge/lima-kilo] (fix_webservice_2) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/169 [14:11:17] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 [14:11:18] 10Data-Services, 10MW-1.43-notes (1.43.0-wmf.15; 2024-07-23), 13Patch-For-Review, 10Wiki-Setup (Create): Create a Wikimedians of United Arab Emirates User Group Wiki - https://phabricator.wikimedia.org/T362529#9985611 (10fnegri) > @ABran-WMF, @fnegri - I believe that we are ready to run the sre.wikireplica... [14:11:35] (03open) 10dcaro: toolforge_deploy_mr: handle tools-webservice packages [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/170 [14:11:55] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 [14:12:09] (03update) 10dcaro: toolforge_deploy_mr: handle tools-webservice packages [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/170 [14:12:24] (03update) 10dcaro: toolforge_deploy_mr: handle tools-webservice packages [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/170 [14:12:39] (03update) 10dcaro: toolforge_deploy_mr: make install/uninstall non-interactive [repos/cloud/toolforge/lima-kilo] (fix_webservice_2) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/169 [14:12:52] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 [14:13:43] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [14:13:51] 06cloud-services-team, 10Toolforge: toolforge: ingress-nginx pods get OOMkilled, consider scaling up - https://phabricator.wikimedia.org/T370162 (10aborrero) 03NEW [14:13:53] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [14:14:26] (03open) 10aborrero: ingress-nginx: scale up deployment [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/422 (https://phabricator.wikimedia.org/T370162) [14:14:43] (03approved) 10aborrero: toolforge_deploy_mr: make install/uninstall non-interactive [repos/cloud/toolforge/lima-kilo] (fix_webservice_2) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/169 (owner: 10dcaro) [14:15:06] (03approved) 10aborrero: toolforge_deploy_mr: handle tools-webservice packages [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/170 (owner: 10dcaro) [14:18:43] (03merge) 10dcaro: toolforge_deploy_mr: handle tools-webservice packages [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/170 [14:18:45] (03update) 10dcaro: toolforge_deploy_mr: make install/uninstall non-interactive [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/169 [14:19:18] (03merge) 10dcaro: toolforge_deploy_mr: make install/uninstall non-interactive [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/169 [14:19:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25 - https://phabricator.wikimedia.org/T316107#9985684 (10aborrero) 05Open→03Resolved [14:21:01] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: upgrade worker nodes to k8s 1.25 - https://phabricator.wikimedia.org/T369171#9985680 (10aborrero) 05In progress→03Resolved a:03aborrero [14:28:33] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [14:28:35] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [14:28:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:28:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:48:38] 10Tool-paulina: Customize the work page according to the type of work - https://phabricator.wikimedia.org/T370166 (10Pepe_piton) 03NEW [14:53:23] 10Tool-paulina: Customize the work page according to the type of work - https://phabricator.wikimedia.org/T370166#9985981 (10Pepe_piton) [14:58:26] (03update) 10dcaro: ingress-nginx: scale up deployment [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/422 (https://phabricator.wikimedia.org/T370162) (owner: 10aborrero) [14:58:56] (03open) 10sstefanova: Add new flavor for 'integration' project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/15 (https://phabricator.wikimedia.org/T370127) [14:59:13] (03approved) 10dcaro: ingress-nginx: scale up deployment [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/422 (https://phabricator.wikimedia.org/T370162) (owner: 10aborrero) [15:03:15] (03update) 10sstefanova: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) (owner: 10dcaro) [15:03:33] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [15:03:44] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [15:05:42] 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations, 10Puppet CI: Cloud VPS "puppet-diffs" project Buster deprecation - https://phabricator.wikimedia.org/T367547#9986045 (10Andrew) *nudge* [15:08:23] 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Request new flavor for integration project - https://phabricator.wikimedia.org/T370127#9986069 (10Slst2020) 05Open→03In progress [15:09:32] (03approved) 10dcaro: builds-api: bump to 0.0.163-20240716083349-0dc2fa69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/418 (https://phabricator.wikimedia.org/T367181) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [15:09:34] (03update) 10dcaro: builds-api: bump to 0.0.163-20240716083349-0dc2fa69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/418 (https://phabricator.wikimedia.org/T367181) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [15:09:36] (03merge) 10dcaro: builds-api: bump to 0.0.163-20240716083349-0dc2fa69 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/418 (https://phabricator.wikimedia.org/T367181) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [15:10:41] 10Tool-paulina: Customize the work page according to the type of work - https://phabricator.wikimedia.org/T370166#9986083 (10Pepe_piton) p:05Triage→03Medium [15:11:02] (03update) 10dcaro: api: consolidate paths [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T365014) (owner: 10sstefanova) [15:12:10] (03approved) 10dcaro: Add new flavor for 'integration' project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/15 (https://phabricator.wikimedia.org/T370127) (owner: 10sstefanova) [15:12:20] (03update) 10dcaro: Add new flavor for 'integration' project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/15 (https://phabricator.wikimedia.org/T370127) (owner: 10sstefanova) [15:13:51] (03merge) 10sstefanova: Add new flavor for 'integration' project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/15 (https://phabricator.wikimedia.org/T370127) [15:28:21] (03approved) 10dcaro: api endpoints: use plural paths [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/51 (https://phabricator.wikimedia.org/T365014) (owner: 10sstefanova) [15:28:24] (03update) 10dcaro: api endpoints: use plural paths [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/51 (https://phabricator.wikimedia.org/T365014) (owner: 10sstefanova) [15:28:37] (03approved) 10dcaro: api: rename api resources to plural [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T365014) (owner: 10sstefanova) [15:32:31] 10Cloud-VPS (Quota-requests): Request new flavor for integration project - https://phabricator.wikimedia.org/T370127#9986209 (10Slst2020) Your brand new flavor should be available now. Enjoy! [15:32:40] 10Cloud-VPS (Quota-requests): Request new flavor for integration project - https://phabricator.wikimedia.org/T370127#9986210 (10Slst2020) 05In progress→03Resolved [15:34:07] (03update) 10sstefanova: api: consolidate paths [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T365014) [15:34:11] (03approved) 10dcaro: ci: remove unused variable MEMORY_OPTIMIZED [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/104 [15:34:13] (03update) 10dcaro: ci: remove unused variable MEMORY_OPTIMIZED [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/104 [15:34:16] (03merge) 10dcaro: ci: remove unused variable MEMORY_OPTIMIZED [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/104 [15:34:22] (03merge) 10sstefanova: api: consolidate paths [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T365014) [15:35:08] (03approved) 10dcaro: remove /api prefix [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/50 (owner: 10sstefanova) [15:36:29] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.319-20240716153429-ac8e3c99 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/423 (https://phabricator.wikimedia.org/T363346) [15:37:43] (03update) 10sstefanova: api: consolidate paths [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T365014) [15:40:00] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.164-20240716153428-d1c47de5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/424 [15:42:25] 10Tool-paulina: Add ethnic group to author's page - https://phabricator.wikimedia.org/T370173 (10Pepe_piton) 03NEW [15:46:18] 10Tool-paulina: Add ethnic group to author's page - https://phabricator.wikimedia.org/T370173#9986339 (10Pepe_piton) p:05Triage→03Medium a:03Pepe_piton [15:48:17] (03update) 10aborrero: ingress-nginx: scale up deployment [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/422 (https://phabricator.wikimedia.org/T370162) [15:57:57] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.2 Create "standard" tool to measure the number of steps for a deployment - https://phabricator.wikimedia.org/T368602#9986403 (10dcaro) [15:58:02] (03update) 10aborrero: ingress-nginx: scale up deployment [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/422 (https://phabricator.wikimedia.org/T370162) [15:58:27] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.2 Create "standard" tool to measure the number of steps for a deployment - https://phabricator.wikimedia.org/T368602#9986404 (10dcaro) I copied and re-ordered the points to the task so we can start wor... [16:00:01] 10Toolforge (Toolforge iteration 12): [sct.backend] create skeleton fastapi API - https://phabricator.wikimedia.org/T370176 (10dcaro) 03NEW [16:01:46] 10Toolforge (Toolforge iteration 12): [sct.backend] create skeleton fastapi API - https://phabricator.wikimedia.org/T370176#9986425 (10dcaro) [16:01:57] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment - https://phabricator.wikimedia.org/T368602#9986427 (10dcaro) [16:02:21] 10Toolforge (Toolforge iteration 12): [sct.frontend] Create skeleton Vue.js application skeleton - https://phabricator.wikimedia.org/T370178 (10dcaro) 03NEW [16:03:39] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 12), 07Epic: [Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment - https://phabricator.wikimedia.org/T368602#9986460 (10dcaro) [16:03:41] 10Toolforge (Toolforge iteration 12): [sct.backend] create skeleton fastapi API - https://phabricator.wikimedia.org/T370176#9986462 (10dcaro) [16:04:01] 10Toolforge (Toolforge iteration 12): [sct.backend] Create skeleton fastapi API - https://phabricator.wikimedia.org/T370176#9986464 (10dcaro) [16:04:46] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] (bump_jobs-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [16:04:53] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] (bump_jobs-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [16:09:58] (03approved) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] (bump_jobs-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [16:09:59] (03update) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] (bump_jobs-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [16:10:02] (03merge) 10dcaro: functional.direct-api: add openapi checks for each api [repos/cloud/toolforge/toolforge-deploy] (bump_jobs-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/411 (https://phabricator.wikimedia.org/T367181) [16:10:04] (03update) 10dcaro: jobs-api: bump to 0.0.319-20240716153429-ac8e3c99 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/423 (https://phabricator.wikimedia.org/T363346 https://phabricator.wikimedia.org/T367181) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:20:31] 10Toolforge (Toolforge iteration 12): [sct.frontend] Create skeleton Vue.js application skeleton - https://phabricator.wikimedia.org/T370178#9986560 (10dcaro) p:05Triage→03High [16:20:46] 10Toolforge (Toolforge iteration 12): [sct.frontend] Create skeleton Vue.js application skeleton - https://phabricator.wikimedia.org/T370178#9986564 (10dcaro) a:05dcaro→03None [16:21:16] 10Toolforge (Toolforge iteration 12): [sct.backend] Create skeleton fastapi API - https://phabricator.wikimedia.org/T370176#9986561 (10dcaro) p:05Triage→03High a:05dcaro→03None [16:22:46] 10Tool-paulina: Improve country page - https://phabricator.wikimedia.org/T370179 (10Pepe_piton) 03NEW [16:23:03] 10Tool-paulina: Improve country page - https://phabricator.wikimedia.org/T370179#9986581 (10Pepe_piton) a:03Pepe_piton [16:23:23] (03open) 10dcaro: pre-commit: add openapi version bump check [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/105 (https://phabricator.wikimedia.org/T356972) [16:23:36] 10Tool-paulina: Improve country page - https://phabricator.wikimedia.org/T370179#9986585 (10Pepe_piton) p:05Triage→03Medium [16:29:38] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [builds-api,envvars-api,jobs-api] bump the version in the openapi definition when bumping the package version - https://phabricator.wikimedia.org/T356972#9986609 (10dcaro) 05Resolved→03In progress [16:29:56] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [builds-api,envvars-api,jobs-api] bump the version in the openapi definition when bumping the package version - https://phabricator.wikimedia.org/T356972#9986615 (10dcaro) Turns out I forgot to send an MR for jobs-api (did push a branch though xd) [16:33:18] (03approved) 10dcaro: ci: force memory-optimized runners [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/41 [16:33:23] (03merge) 10dcaro: ci: force memory-optimized runners [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/41 [16:36:44] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.55-20240716163331-6f3efd6d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/425 [16:36:45] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [16:36:55] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [16:43:24] 10Cloud-Services, 10Catalyst: Moving proxies across wmcs projects for patchdemo.wmflabs.org - https://phabricator.wikimedia.org/T370080#9986739 (10matmarex) I was going to use the migration between the Cloud VPS projects to also migrate from .wmflabs.org to .wmcloud.org. I've already set up a proxy using the n... [16:49:35] (03open) 10dcaro: run_functional_tests: enable running as a different user [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/426 [16:50:22] (03update) 10dcaro: run_functional_tests: enable running as a different user [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/426 [16:56:04] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [16:56:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:56:17] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [16:56:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:00:54] 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations, 10Puppet CI: Cloud VPS "puppet-diffs" project Buster deprecation - https://phabricator.wikimedia.org/T367547#9986837 (10jhathaway) thanks, still on my docket for this week [17:05:27] (03update) 10dcaro: run_functional_tests: enable running as a different user [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/426 [17:10:34] 10Toolforge (Toolforge iteration 12): [cli] the generic cli swallows the `--` from other commands - https://phabricator.wikimedia.org/T370184 (10dcaro) 03NEW [17:10:55] (03update) 10dcaro: run_functional_tests: enable running as a different user [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/426 [17:22:10] (03update) 10dcaro: run_functional_tests: enable running as a different user [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/426 [17:26:30] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [17:26:32] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [17:26:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:26:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:33:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [17:37:26] 10Horizon: Horizon does not commit changes to cloud/instance-puppet git repo since June 24th 2024 - https://phabricator.wikimedia.org/T370136#9986988 (10bd808) [17:38:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [18:31:14] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering: Increased visibility in wiki-replicas for volunteers fighting vandals - https://phabricator.wikimedia.org/T284944#9987174 (10sguebo_WMF) Hi @joanna_borun, tagging you here as per @acooper’s guidance. The current task is a co... [18:42:33] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T370114#9987320 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/441 [18:42:35] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T370114#9987321 (10rook) 05Open→03Resolved a:03rook [18:42:49] vivian-rook closed https://github.com/toolforge/paws/pull/441 [18:50:32] 10Cloud-VPS (Quota-requests): Request to increase catalyst project: cores and memory (2024-07-16) - https://phabricator.wikimedia.org/T370195 (10matmarex) 03NEW [18:52:00] (03PS1) 10Krinkle: docs: Write "History" section for the README [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1054644 [18:56:14] (03PS2) 10Krinkle: docs: Write "History" section for the README [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1054644 [18:56:46] (03PS3) 10Krinkle: docs: Write "History" section for the README [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1054644 [18:57:49] (03PS4) 10Krinkle: docs: Write "History" section for the README [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1054644 [19:08:32] (03PS1) 10Krinkle: docs: Move intro from API.md to README.md [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1054648 [19:08:51] (03PS2) 10Krinkle: docs: Move intro from API.md to README.md [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1054648 [19:09:03] (03CR) 10Krinkle: [C:03+2] docs: Move intro from API.md to README.md [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1054648 (owner: 10Krinkle) [19:09:34] (03Merged) 10jenkins-bot: docs: Move intro from API.md to README.md [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1054648 (owner: 10Krinkle) [19:10:15] (03PS1) 10Krinkle: docs: Add link to primary location [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1054651 [19:10:20] (03CR) 10Krinkle: [C:03+2] docs: Add link to primary location [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1054651 (owner: 10Krinkle) [19:11:03] (03Merged) 10jenkins-bot: docs: Add link to primary location [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1054651 (owner: 10Krinkle) [19:11:07] (03PS5) 10Krinkle: docs: Write "History" section for the README [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1054644 [19:12:00] (03PS6) 10Krinkle: docs: Write a little "History" section in the README [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1054644 [19:12:03] (03CR) 10Krinkle: [C:03+2] docs: Write a little "History" section in the README [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1054644 (owner: 10Krinkle) [19:12:31] (03Merged) 10jenkins-bot: docs: Write a little "History" section in the README [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1054644 (owner: 10Krinkle) [19:52:54] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [19:52:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:53:06] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [19:53:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:09:41] 10Tool-schedule-deployment: Display link to wikitext diff when reporting a successful patch addition - https://phabricator.wikimedia.org/T367948#9987797 (10bd808) This will need some template string escaping work to implement. The current render templates for [[https://flask.palletsprojects.com/en/3.0.x/patterns... [20:10:02] 10Tool-schedule-deployment: Display link to wikitext diff when reporting a successful patch addition - https://phabricator.wikimedia.org/T367948#9987798 (10bd808) p:05Triage→03Medium [20:10:18] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[35-38] - https://phabricator.wikimedia.org/T363344#9987799 (10Jclark-ctr) @VRiley-WMF if you can update with 2nd network connection then hand over to @cmooney [20:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:23:22] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [20:23:24] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [20:23:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:23:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:53:15] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: Fix 'openstack database instance rebuild' - https://phabricator.wikimedia.org/T355721#9988013 (10Andrew) I have a clue! I think that rebuild misbehaves when there are active snapshots of a VM. On my first test, this seems to have made... [21:19:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:43:16] 10Tool-paulina: Link to Cradle to create new item if no results - https://phabricator.wikimedia.org/T370223 (10Pepe_piton) 03NEW [21:47:08] 10Tool-paulina: Link to Cradle to create new item if no results - https://phabricator.wikimedia.org/T370223#9988268 (10Pepe_piton) a:03Pepe_piton [21:55:00] 10Tool-yearinreview, 07good first task: Please attribute the original, add disclaimer and add the LICENSE - https://phabricator.wikimedia.org/T366114#9988285 (10Jdlrobson) Thanks for prioritizing this @Gopavasanth ! Looks great! Regarding Gitlab, all my projects are on Github and Gerrit and there is quite... [21:59:15] 10Tool-paulina: Link to Cradle to create new item if no results - https://phabricator.wikimedia.org/T370223#9988297 (10Pepe_piton) p:05Triage→03Medium [22:16:09] 10Tool-paulina: Set up of local versions - https://phabricator.wikimedia.org/T370226 (10Pepe_piton) 03NEW [22:17:49] 10Tool-paulina: Set up of local versions - https://phabricator.wikimedia.org/T370226#9988375 (10Pepe_piton) p:05Triage→03High a:03Pepe_piton [22:50:07] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [22:50:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:50:19] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [22:50:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:21:03] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [23:21:05] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [23:21:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:21:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:28:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [23:33:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [23:39:10] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Request to increase catalyst project: cores and memory (2024-07-16) - https://phabricator.wikimedia.org/T370195#9988714 (10bd808) +1