[03:44:45] (SystemdUnitFailed) firing: (7) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:04:45] (SystemdUnitFailed) firing: (8) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:21:14] 10SRE-tools, 10Dumps-Generation, 10Infrastructure-Foundations: custom partman recipe dumpsdata100X-no-data-format.cfg causes installer to hang at partitioning menu - https://phabricator.wikimedia.org/T339929 (10ArielGlenn) Found the --no-pxe option so this host has been reimaged. Besides the partition manage... [07:49:45] (SystemdUnitFailed) firing: (8) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:54:00] releng asked for creation of a special sysuser in LDAP, my proposal for how to handle such requests is at https://phabricator.wikimedia.org/T338950#8951946, input welcome here or on task! [09:01:47] moritzm: what you wrot sounds good to me [09:06:06] 10SRE-tools, 10Infrastructure-Foundations: Read Ganeti cluster config for cookboks from Netbox - https://phabricator.wikimedia.org/T340015 (10MoritzMuehlenhoff) [09:08:29] ack, thanks [11:49:45] (SystemdUnitFailed) firing: (7) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:02:09] 10SRE-tools, 10Infrastructure-Foundations: Read Ganeti cluster config for cookbooks from Netbox - https://phabricator.wikimedia.org/T340015 (10Reedy) [13:23:43] jbond / volans if you have the time, Netbox OIDC patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/922506 [13:28:57] slyngs: great, thanks! I've left one question [13:31:06] Oh, and a good question :-) [13:47:32] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10Patch-For-Review: ServiceLVS without monitor breaks spicerack - https://phabricator.wikimedia.org/T339243 (10Clement_Goubert) Release has been cut, cookbook works ok now. @Volans do you want to keep the task open to keep looking for a way to enforce... [13:49:11] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10Patch-For-Review: ServiceLVS without monitor breaks spicerack - https://phabricator.wikimedia.org/T339243 (10Volans) I was about to come here and comment as I just deployed spicerack to production :) I think we can keep it open for now to see if we... [14:49:13] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) [15:20:01] 10SRE-tools, 10Infrastructure-Foundations: Package pyGNMI and dictdiff to be used by cookbooks - https://phabricator.wikimedia.org/T340045 (10ayounsi) [15:20:11] 10SRE-tools, 10Infrastructure-Foundations: Package pyGNMI and dictdiffer to be used by cookbooks - https://phabricator.wikimedia.org/T340045 (10ayounsi) [15:35:41] 10SRE-tools, 10DBA, 10Infrastructure-Foundations: Create a cookbook for cloning a mariadb database into another - https://phabricator.wikimedia.org/T340048 (10Ladsgroup) [15:35:57] 10SRE-tools, 10DBA, 10Infrastructure-Foundations: Create a cookbook for cloning a mariadb database into another - https://phabricator.wikimedia.org/T340048 (10Ladsgroup) p:05Triage→03Medium [15:42:31] 10SRE-tools, 10Infrastructure-Foundations: Package pyGNMI and dictdiffer to be used by cookbooks - https://phabricator.wikimedia.org/T340045 (10MoritzMuehlenhoff) I'll look into dictdiffer [15:49:45] (SystemdUnitFailed) firing: (7) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:49:45] (SystemdUnitFailed) firing: (7) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:19:45] (SystemdUnitFailed) firing: (8) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:19:45] (SystemdUnitFailed) firing: (8) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:46:07] 10netbox, 10Infrastructure-Foundations, 10decommission-hardware, 10ops-eqiad, 10serviceops-collab: decommission gerrit1001.wikimedia.org (dcops, netbox) - https://phabricator.wikimedia.org/T340077 (10Dzahn) [22:47:26] 10netbox, 10Infrastructure-Foundations, 10decommission-hardware, 10ops-eqiad, 10serviceops-collab: decommission gerrit1001.wikimedia.org (dcops, netbox) - https://phabricator.wikimedia.org/T340077 (10Dzahn) @Volans Could you delete / sync the special entry in netbox for former gerrit IP 208.80.154.137 (a... [22:48:45] 10netbox, 10Infrastructure-Foundations, 10decommission-hardware, 10ops-eqiad, 10serviceops-collab: decommission gerrit1001.wikimedia.org (dcops, netbox) - https://phabricator.wikimedia.org/T340077 (10Dzahn) a:05Dzahn→03None [22:49:56] 10netbox, 10Infrastructure-Foundations, 10decommission-hardware, 10ops-eqiad, 10serviceops-collab: decommission gerrit1001.wikimedia.org (dcops, netbox) - https://phabricator.wikimedia.org/T340077 (10Dzahn) p:05Triage→03Medium