[01:09:45] (SystemdUnitFailed) firing: update-ubuntu-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:09:45] (SystemdUnitFailed) firing: update-ubuntu-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:09:45] (SystemdUnitFailed) resolved: update-ubuntu-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:22:09] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) [11:01:01] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Puppet (Puppet 7.0): PKI: add the new puppet CA to the pki infrastructre - https://phabricator.wikimedia.org/T340557 (10jbond) 05Open→03In progress p:05Triage→03Medium [11:48:32] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10Puppet (Puppet 7.0): PKI: add the new puppet CA to the pki infrastructre - https://phabricator.wikimedia.org/T340557 (10jbond) 05In progress→03Resolved a:03jbond This has now been corrected systems on the new puppet infrastruct... [12:58:08] 10CFSSL-PKI, 10Cloud-VPS, 10Infrastructure-Foundations: Simple multiroot cfssl PKI setup for Cloud-VPS projects - https://phabricator.wikimedia.org/T340742 (10taavi) [13:39:08] 10CAS-SSO, 10Data-Catalog, 10Data-Engineering, 10Data-Platform-SRE, 10Infrastructure-Foundations: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10JArguello-WMF) [13:49:45] (SystemdUnitFailed) firing: wmf_auto_restart_squid.service Failed on urldownloader2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:35:55] 10netops, 10Analytics-Radar, 10Data-Engineering, 10Infrastructure-Foundations: Errors for ifup@ens5.service after rebooting Ganeti VMs - https://phabricator.wikimedia.org/T273026 (10Aklapper) (Adding #Data-Engineering project tag for re-triage, as #Analytics-Radar is an inactive project tag after #Analytic... [15:23:43] 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations: Cookbooks for Ganeti maintenance tasks - https://phabricator.wikimedia.org/T283319 (10MoritzMuehlenhoff) [15:23:53] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Spicerack, and 4 others: Convert makevm to spicerack cookbook - https://phabricator.wikimedia.org/T203963 (10MoritzMuehlenhoff) [15:24:14] 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations, 10SRE, and 2 others: Create a spicerack cookbook to empty a ganeti node from VMs - https://phabricator.wikimedia.org/T203964 (10MoritzMuehlenhoff) 05Open→03Resolved This has been implemented with the new sre.ganeti.drain-node cookbook, which I've use... [15:43:12] 10CAS-SSO, 10Data-Engineering, 10Data-Platform-SRE, 10Infrastructure-Foundations: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10JArguello-WMF) [16:09:45] (SystemdUnitFailed) firing: (2) wmf_auto_restart_squid.service Failed on urldownloader2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:19:45] (SystemdUnitFailed) resolved: (2) wmf_auto_restart_squid.service Failed on urldownloader2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:20:17] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10Puppet (Puppet 7.0): PKI: add the new puppet CA to the pki infrastructre - https://phabricator.wikimedia.org/T340557 (10jbond) 05Resolved→03Open Unfortunately i closed this too soon. things work fine on the puppetserver but now... [16:21:22] hi got a bit of a strange issue with a combination of puppet, go and pki. i have documented it https://phabricator.wikimedia.org/T340557 would appreciated if someone could take a second look to make sure im not missing something obvious and perhpas find the error :) [16:21:35] specifically https://phabricator.wikimedia.org/T340557#8977097 [21:13:32] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Resource attributes are quoted inconsistently - https://phabricator.wikimedia.org/T91908 (10Dzahn) The only thing that changed since 2015 is the numbers are now higher :) ` ~/repos/puppet$ git grep "ensure *=> *\([a-z]\+\)" | wc -l 3027... [23:43:06] 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10JArguello-WMF)