[00:53:03] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12052339 (10BTullis) [00:53:58] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12052340 (10BTullis) [06:12:25] FIRING: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:17:25] RESOLVED: SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:17:31] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Reimaging with a cookbook this kind of server with a wrong management password leads to exception - https://phabricator.wikimedia.org/T384462#12053382 (10CWilliams-WMF) [10:21:42] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Reimaging with a cookbook this kind of server with a wrong management password leads to exception - https://phabricator.wikimedia.org/T384462#12053392 (10CWilliams-WMF) @Volans I have linked the ticket that I created for a very similar scenario, now mar... [10:31:05] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Reimaging with a cookbook this kind of server with a wrong management password leads to exception - https://phabricator.wikimedia.org/T384462#12053465 (10MoritzMuehlenhoff) This should soon no longer be an issue once https://phabricator.wikimedia.org/T4... [11:00:39] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Reimaging with a cookbook this kind of server with a wrong management password leads to exception - https://phabricator.wikimedia.org/T384462#12053564 (10CWilliams-WMF) @MoritzMuehlenhoff thanks for that! [13:01:25] FIRING: SystemdUnitFailed: netbox_ganeti_codfw_test_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:16:25] RESOLVED: SystemdUnitFailed: netbox_ganeti_codfw_test_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:44:42] moritzm, XioNoX - o/ should we do https://gerrit.wikimedia.org/r/c/operations/puppet/+/1294952 next week? [13:45:33] sure [13:47:18] sounds good [14:11:37] the only thing to keep in mind is that the krb hosts are not uefi (so we may want to change them) and the current partman recipe doesn't preserve /srv [14:17:01] it's a server which is already running since 2 years, let's simply keep it on BIOS [14:17:26] and we do have reuse-raid1-2dev.cfg for BIOS [14:19:01] but we probably wouldn't really again anything with it? /srv only has the keytabs which were generated and those are all in the private repo anyway [14:19:25] while the crucial runtime date is in /var/lib/krb5kdc [15:02:47] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12054668 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=2186f648-4746-4ad9-8401-8db932f31c67) set by pt1979@cumin2003 for 2:00:00 on 2 host(s) and their services with reason: j... [15:02:55] should we save that part then? Or do you think that we can start from empty? [15:50:25] both approaches would work, let's have a closer look on the pros and cons on Monday before we pick a migration path [15:51:20] if we fully reimage and go the kprop venue would could also reimage krb1002 to trixie, those KDCs need to be updated at some point anyway [16:29:29] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12055347 (10Papaul) [18:55:25] FIRING: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:00:25] RESOLVED: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed