[02:53:22] <jinxer-wm>	 (SystemdUnitFailed) firing: update-ubuntu-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:47:23] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10User-jbond: Investigate the potential benefits of BGPalerter - https://phabricator.wikimedia.org/T230600 (10fgiunchedi) I noticed the weekly "software-update" emails from bgpalerter, can those be disabled ? (i.e. the version check I guess)
[06:54:41] <jinxer-wm>	 (SystemdUnitFailed) firing: update-ubuntu-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:20:32] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10User-jbond: Investigate the potential benefits of BGPalerter - https://phabricator.wikimedia.org/T230600 (10ayounsi) Relevant https://github.com/nttgin/BGPalerter/issues/1058
[08:52:51] <jinxer-wm>	 (SystemdUnitFailed) resolved: update-ubuntu-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:52:51] <jinxer-wm>	 (SystemdUnitFailed) firing: prometheus_puppet_agent_stats.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:02:51] <jinxer-wm>	 (SystemdUnitFailed) resolved: prometheus_puppet_agent_stats.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:03:08] <moritzm>	 ^ puppetdb2003 is me doing some tests
[13:50:28] <wikibugs>	 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10tox-wikimedia, and 2 others: Introduce Python code formatters usage - https://phabricator.wikimedia.org/T211750 (10jbond)
[14:38:25] <wikibugs>	 10puppet-compiler, 10Infrastructure-Foundations: puppet compiler shows diffs after a host has been rebuild - https://phabricator.wikimedia.org/T334478 (10jbond) p:05Triage→03Medium
[14:38:34] <wikibugs>	 10puppet-compiler, 10Infrastructure-Foundations: puppet compiler shows diffs after a host has been rebuild - https://phabricator.wikimedia.org/T334478 (10jbond) p:05Medium→03Low
[14:45:40] <wikibugs>	 10puppet-compiler, 10Infrastructure-Foundations: puppet compiler shows diffs after a host has been rebuild - https://phabricator.wikimedia.org/T334478 (10ssingh) Thanks for filing this and writing it down! PCC for posterity: https://puppet-compiler.wmflabs.org/output/907897/40592/
[15:40:58] <wikibugs>	 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10LSobanski)
[16:00:10] <wikibugs>	 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10LSobanski) p:05Triage→03Medium
[16:08:13] <wikibugs>	 10netops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10Jelto)
[17:19:57] <wikibugs>	 10netbox, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: Issues converting services from active/passive to active/active - https://phabricator.wikimedia.org/T330084 (10jbond) from @brandon via irc   >it *seems* like that error in the ticket would've only happened if the puppet agent...
[17:22:51] <jinxer-wm>	 (SystemdUnitFailed) firing: prometheus-puppet-ca-exporter.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:27:51] <jinxer-wm>	 (SystemdUnitFailed) resolved: prometheus-puppet-ca-exporter.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[19:16:40] <wikibugs>	 10Mail, 10fundraising-tech-ops: DMarc Email Address for Wikimedia.org - https://phabricator.wikimedia.org/T316899 (10XenoRyet)
[23:02:20] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10Traffic: Adjust routing policy to increase SSH session speed from East Asia to toolforge - https://phabricator.wikimedia.org/T334530 (10Stang)