[03:05:19] (SystemdUnitFailed) firing: (4) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:10:19] (SystemdUnitFailed) firing: (4) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:50:57] 10SRE-tools, 10Infrastructure-Foundations, 10Traffic: Cookbook to depool a site in AuthDNS - https://phabricator.wikimedia.org/T334048 (10ayounsi) My suggestion to use a cookbook is because it's what SREs are familiar with, centralized in one place, can be nested for larger scope automation, provide abstract... [07:10:13] (SystemdUnitFailed) firing: (4) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:50:13] (SystemdUnitFailed) firing: (7) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:57:05] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review: puppet: augeas and augeasprovider both define the augeas feature - https://phabricator.wikimedia.org/T335572 (10jbond) [11:35:36] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10fgiunchedi) [12:04:06] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10MoritzMuehlenhoff) [12:30:28] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review: puppet: augeas and augeasprovider both define the augeas feature - https://phabricator.wikimedia.org/T335572 (10jbond) 05Open→03Stalled The patch has fixed the issues for now, ill wait to see what upstream says [13:50:13] (SystemdUnitFailed) firing: (7) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:53:30] 10netops, 10Infrastructure-Foundations, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [cloudvirt] Move to jumbo frames - https://phabricator.wikimedia.org/T330075 (10nskaggs) [15:18:35] 10SRE-tools, 10Infrastructure-Foundations: redfish: minimum version support - https://phabricator.wikimedia.org/T328593 (10Papaul) @jbond as for 10 Mars 2023 the IDRAC latest version for PowerEdge R430 is 2.84 or to be able to run the firmware cookbook Redfish wants the idrac to be at minimum 3.30 . So I thi... [17:50:13] (SystemdUnitFailed) firing: (7) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:40:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:40:13] (SystemdUnitFailed) firing: (8) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:33:17] 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10colewhite) [23:51:13] (DiskSpace) firing: Disk space puppetmaster1001:9100:/ 5.95% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace