[01:54:20] FIRING: SystemdUnitFailed: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:04:20] RESOLVED: SystemdUnitFailed: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:05:29] 10netbox, 06Infrastructure-Foundations: Change icinga link to alerts.w.o in netbox device page - https://phabricator.wikimedia.org/T371079 (10fgiunchedi) 03NEW [08:13:04] hello folks! I am going to reimage sretest1001 to test the cookbook with the tftp-only dhcp option [10:00:41] quick code review for the reimage cookbook if anybody has time: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1057180 [10:58:05] 10netbox, 06Infrastructure-Foundations: Change icinga link to alerts.w.o in netbox device page - https://phabricator.wikimedia.org/T371079#10017576 (10cmooney) p:05Triage→03Medium Thanks @fgiunchedi yeah the request makes sense. I think we can inlcude whatever links we wish, so it would also be possible t... [12:05:42] 10netbox, 06Infrastructure-Foundations: Change icinga link to alerts.w.o in netbox device page - https://phabricator.wikimedia.org/T371079#10017705 (10fgiunchedi) Thanks for the feedback @cmooney ! Absolutely no rush / timeline for this change re: netbox 4 upgrade. alerts.w.o can and does display icinga alerts... [12:59:35] topranks: debian install for sretest1001 completed fine with the dhcp tftp boot settings! [12:59:56] the cookbook is finishing but the install step is over [13:00:03] I'll report everything in the task [13:00:08] elukey: woohoo! [13:00:22] so now just adding a flag to the reimage cookbook should be sufficient [13:00:28] Nice work, definitely good to have that option to relieve the pressure [13:00:41] exactly yes [13:01:07] and with the new spicerack's dhcp module it should be easier to test EFI (in theory) [13:01:28] nice way to be finishing up the week :) [13:24:07] cdanis: o/ lemme know if you have time for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1054894, to get on the same page (so I'll try to finish the work before next monday's team meeting) [13:25:57] elukey: lgtm <3 [13:26:20] ah wow super quick! thanks! [13:26:37] I'll rebase it since Tiziano (new SRE in olly) was added to sre-admins [13:26:46] we need to stop calling it that btw [13:26:51] it's a bad name [13:27:03] oh yes when ops-limited is rolled out I'll remove sre-admins [13:27:17] 🫡 [13:27:17] update the docs etc.. [13:40:55] added a write-up of what happened earlier on with puppet private in https://phabricator.wikimedia.org/T368023#10017817 [13:41:11] rolling back the dumps_cloud_ip_ranges timer now, one server at the time [13:41:26] it runs at midnight so there shouldn't be any issue, but.. [16:00:54] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Spicerack: expand Supermicro support in the Redfish module - https://phabricator.wikimedia.org/T365372#10018565 (10elukey) Next steps: * Add mac address field to https://netbox.wikimedia.org/extras/scripts/provision_ser... [19:45:06] 10Mail, 06Infrastructure-Foundations, 06SRE: postfix mx puppetry - https://phabricator.wikimedia.org/T325395#10019371 (10jhathaway)