[00:03:27] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T378158#10261317 (10LibUp-bot) [01:21:30] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:36:01] (03CR) 10Blake Hale: "recheck" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1079381 (https://phabricator.wikimedia.org/T373708) (owner: 10Krinkle) [05:20:05] 10Tool-Gerrit-Patch-Uploader: Add patch creation interface - https://phabricator.wikimedia.org/T378165 (10Krinkle) 03NEW [05:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:54:25] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: tofu-infra: add support for DNS zones created by wmfkeystonehook - https://phabricator.wikimedia.org/T376110#10261854 (10aborrero) Turns out, we cannot avoid with opentofu the DNS zone transfer dancing required when creating a subdomain of a zone declar... [09:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:16:39] 06cloud-services-team, 10Toolforge: toolforge: integrate fourohfour as a custom component, rather than a normal tool - https://phabricator.wikimedia.org/T369364#10262045 (10Raymond_Ndibe) Since the redis cache here doesn't need to be persisted, anyone see any problem with having the redis cache be either a dif... [10:44:18] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [10:53:04] 06cloud-services-team, 10Toolforge: toolforge: integrate fourohfour as a custom component, rather than a normal tool - https://phabricator.wikimedia.org/T369364#10262117 (10aborrero) >>! In T369364#10262045, @Raymond_Ndibe wrote: > Since the redis cache here doesn't need to be persisted, anyone see any problem... [10:57:51] 10Toolforge (Toolforge iteration 16): [lima-kilo] support caching of container images using a cache disk - https://phabricator.wikimedia.org/T378180 (10Raymond_Ndibe) 03NEW [10:58:56] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [lima-kilo] allow for the creation of a multi-node high availability cluster - https://phabricator.wikimedia.org/T374585#10262119 (10Raymond_Ndibe) 05Open→03In progress [10:59:10] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [11:05:36] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [11:12:39] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 [11:37:25] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10262201 (10aborrero) I believe I found the reason of the traceback. The designate auth used by neutron is not capable of operating on DNS zones outside of th... [11:42:21] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (https://phabricator.wikimedia.org/T378180) [11:43:54] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 (https://phabricator.wikimedia.org/T378180) [11:45:08] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (https://phabricator.wikimedia.org/T378180) [11:46:06] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [11:46:16] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10262215 (10aborrero) as a way to test this theory, I will be using this patch in cloudcontrol2004-dev: `lang=diff @@ -43,10 +43,10 @@ CONF, 'des... [11:46:51] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [11:47:35] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 (https://phabricator.wikimedia.org/T378180) [11:47:57] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 (https://phabricator.wikimedia.org/T378180) [11:48:07] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 (https://phabricator.wikimedia.org/T378180) [11:48:54] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (https://phabricator.wikimedia.org/T378180) [11:54:24] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_disk) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [11:55:07] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [11:57:05] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [lima-kilo] support caching of container images using a cache disk - https://phabricator.wikimedia.org/T378180#10262233 (10Raymond_Ndibe) [11:59:34] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [lima-kilo] support caching of container images using a cache disk - https://phabricator.wikimedia.org/T378180#10262232 (10Raymond_Ndibe) 05Open→03In progress [11:59:35] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [maintain-harbor] Move to become a toolforge component - https://phabricator.wikimedia.org/T358225#10262238 (10Raymond_Ndibe) [12:00:04] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868#10262237 (10Raymond_Ndibe) [12:00:29] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [harbor] Do not clean up images currently running in production - https://phabricator.wikimedia.org/T377854#10262243 (10Raymond_Ndibe) a:03Raymond_Ndibe [12:00:37] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): Introduce health checks for Toolforge Jobs Framework cronjobs - https://phabricator.wikimedia.org/T377420#10262244 (10Raymond_Ndibe) a:03Raymond_Ndibe [12:02:12] (03approved) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 (https://phabricator.wikimedia.org/T378180) [12:02:19] (03update) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 (https://phabricator.wikimedia.org/T378180) [12:02:26] (03merge) 10raymond-ndibe: [lima-kilo] fix start-devenv.sh bug [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 (https://phabricator.wikimedia.org/T378180) [12:02:27] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (https://phabricator.wikimedia.org/T378180) [12:02:44] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10262235 (10Raymond_Ndibe) 05Open→03In progress [12:04:22] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [maintain-harbor] Move to become a toolforge component - https://phabricator.wikimedia.org/T358225#10262240 (10Raymond_Ndibe) 05Open→03In progress [13:01:30] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10262301 (10aborrero) I could not make it work with that diff. My next theory is that neutron is not even reaching out to the designate API for some reason. I... [13:19:30] (03update) 10rook: Update .gitlab-ci.yml file [repos/cloud/paws] - 10https://gitlab.wikimedia.org/repos/cloud/paws/-/merge_requests/1 [13:21:24] 06cloud-services-team, 10Toolforge, 10Tools, 06Data-Engineering, and 3 others: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327#10262357 (10Ottomata) 05Open→03Declined Closing... [13:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:22:01] vivian-rook opened https://github.com/toolforge/paws/pull/455 [13:22:37] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T378158#10262368 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/455 [13:22:52] (03update) 10rook: Update .gitlab-ci.yml file [repos/cloud/paws] - 10https://gitlab.wikimedia.org/repos/cloud/paws/-/merge_requests/1 [13:44:15] (03update) 10rook: Update .gitlab-ci.yml file [repos/cloud/paws] - 10https://gitlab.wikimedia.org/repos/cloud/paws/-/merge_requests/1 [15:04:00] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10262762 (10aborrero) I am running out of ideas for further debugging this. I will undo all the changes for now. [15:19:38] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192 (10aborrero) 03NEW [15:26:01] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10262823 (10aborrero) [15:26:30] 06cloud-services-team, 10Cloud-VPS: openstack: wmf sink: extend it to support IPv6 - https://phabricator.wikimedia.org/T378192#10262826 (10aborrero) p:05Triage→03Medium [15:41:43] (03close) 10aborrero: Draft: lima-kilo: have kind containerd cache directory be stored in the laptop storage [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/116 [17:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:21:26] vivian-rook closed https://github.com/toolforge/paws/pull/455 [21:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:47:37] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in -1m 25s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire