[01:48:41] 06cloud-services-team, 10Cloud-VPS: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion - https://phabricator.wikimedia.org/T379478 (10Reedy) 03NEW [01:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:52:15] 06cloud-services-team, 10Cloud-VPS: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion - https://phabricator.wikimedia.org/T379478#10306934 (10Reedy) [01:57:17] 10Cloud-VPS (Project-requests): Request creation of WikiPeopleStats VPS project - https://phabricator.wikimedia.org/T378042#10306951 (10Reedy) See {T379478} for an issue with people added to this new project [02:11:08] (03open) 10raymond-ndibe: [calico] update kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585 (https://phabricator.wikimedia.org/T362867) [02:11:24] (03update) 10raymond-ndibe: [calico] update kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585 (https://phabricator.wikimedia.org/T362867) [02:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:47:11] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T362867) [02:47:12] !log raymond-ndibe@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 (T362867) [02:47:16] T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867 [02:47:22] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T362867) [03:07:25] (03update) 10raymond-ndibe: [calico] update kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585 (https://phabricator.wikimedia.org/T362867) [03:07:55] (03update) 10raymond-ndibe: [toolforge-deploy] update calico kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585 (https://phabricator.wikimedia.org/T362867) [03:08:08] (03update) 10raymond-ndibe: [toolforge-deploy] update calico kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585 (https://phabricator.wikimedia.org/T362867) [03:09:28] (03open) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [03:10:59] (03update) 10raymond-ndibe: [toolforge-deploy] update calico kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/585 (https://phabricator.wikimedia.org/T362867) [03:19:42] (03update) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [03:49:28] (03open) 10raymond-ndibe: [envvars-api] k8s.io deps to 0.28.14 [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/48 (https://phabricator.wikimedia.org/T362867) [04:09:36] (03open) 10raymond-ndibe: [registry-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/17 (https://phabricator.wikimedia.org/T362867) [04:14:13] (03open) 10raymond-ndibe: [ingress-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T362867) [04:14:24] (03update) 10raymond-ndibe: [ingress-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T362867) [04:21:50] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10307096 (10Raymond_Ndibe) [04:23:17] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10307097 (10Raymond_Ndibe) [04:29:19] 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 10Wikimedia-maintenance-script-run: Unblock stuck global rename of Hex - https://phabricator.wikimedia.org/T379483#10307109 (10Pppery) It's unattached. It shouldn't be renamed. That's what "unattached" means The desired end state is for https://wikitech.w... [04:41:45] 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 10Wikimedia-maintenance-script-run: Unblock stuck global rename of Hex - https://phabricator.wikimedia.org/T379483#10307119 (10DreamRimmer) This means Scott needs to submit a request at [[https://wikitech.wikimedia.org/wiki/Wikitech:Rename_requests|Wikitec... [04:50:15] (03update) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] (fix_calico_kube_version) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [04:51:08] (03update) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [06:54:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:04:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:33:15] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10307151 (10taavi) [07:37:30] 06cloud-services-team, 10Horizon, 07IPv6: horizon: enable IPv6 security group panels - https://phabricator.wikimedia.org/T377339#10307153 (10taavi) [08:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:16:56] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: nova-fullstack: add support for IPv6 - https://phabricator.wikimedia.org/T379356#10307223 (10taavi) [09:25:26] 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 10Wikimedia-maintenance-script-run: Unblock stuck global rename of Hex - https://phabricator.wikimedia.org/T379483#10307227 (10Scott) This isn't blocking the rename, the Wikitech account can be attached to the global SUL for Hex after my account is renamed... [09:49:52] 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 10Wikimedia-maintenance-script-run: Unblock stuck global rename of Hex - https://phabricator.wikimedia.org/T379483#10307242 (10DreamRimmer) I am unable to rename your account from 'Scott' to 'Hex' because the username 'Hex' is still linked to a Wikitech ac... [09:55:22] (03PS1) 10Majavah: Upgrade pypdf [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1089024 [10:10:14] 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 10Wikimedia-maintenance-script-run: Unblock stuck global rename of Hex - https://phabricator.wikimedia.org/T379483#10307255 (10Scott) Ah, okay. It was such a nightmare getting SUL sorted out back in 2013 and now it turns out that it was never truly finish... [11:39:50] 06cloud-services-team, 10Toolforge, 10Pywikibot, 10Pywikibot-Scripts: Implement a webservice at toolforge.org based on create_isbn_edition script - https://phabricator.wikimedia.org/T379488 (10Xqt) 03NEW [11:40:19] 06cloud-services-team, 10Toolforge, 10Pywikibot, 10Pywikibot-Scripts: Implement a webservice at toolforge.org based on create_isbn_edition script - https://phabricator.wikimedia.org/T379488#10307296 (10Xqt) [13:17:01] 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 10Wikimedia-maintenance-script-run: Unblock stuck global rename of Hex - https://phabricator.wikimedia.org/T379483#10307380 (10Zabe) renamed your wikitech account, can you try attaching it to your global account now? [13:52:41] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate deployment-poolcounter06.deployment-prep.eqiad.wmflabs is about to expire in 17d 23h 58m 30s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [15:05:30] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth: Wikitech users being unexpectedly prompted for 2FA tokens - https://phabricator.wikimedia.org/T379398#10307419 (10taavi) [17:05:03] 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 10Wikimedia-maintenance-script-run: Unblock stuck global rename of Hex - https://phabricator.wikimedia.org/T379483#10307536 (10Scott) Done, thank you @Zabe. [17:40:26] 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 10Wikimedia-maintenance-script-run: Unblock stuck global rename of Hex - https://phabricator.wikimedia.org/T379483#10307543 (10JJMC89) 05Open→03Invalid [18:19:37] 10Cloud-Services, 07Documentation: Improve documentation how to set up a project puppetserver - https://phabricator.wikimedia.org/T379501 (10Physikerwelt) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ a... [18:25:46] 06cloud-services-team, 10Cloud-VPS, 07Documentation: Improve documentation how to set up a project puppetserver - https://phabricator.wikimedia.org/T379501#10307596 (10Physikerwelt) [18:34:17] 06cloud-services-team, 10Cloud-VPS, 10Striker: OpenStack project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10307609 (10Physikerwelt) >>! In T379030#10291065, @bd808 wrote: > https://openstack-browser.toolforge.org/project/5238620953c040e7a9effbe47... [19:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks