[00:06:13] <jinxer-wm>	 (DiskSpace) resolved: Disk space puppetmaster1001:9100:/ 5.497% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[01:26:17] <jinxer-wm>	 (NodeTextfileStale) firing: (2) Stale textfile for puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[05:26:17] <jinxer-wm>	 (NodeTextfileStale) firing: (2) Stale textfile for puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[09:26:17] <jinxer-wm>	 (NodeTextfileStale) firing: (2) Stale textfile for puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[09:56:02] <jinxer-wm>	 (NodeTextfileStale) firing: (2) Stale textfile for puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[12:27:10] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Netbox PuppetDB Import Script Failing for cloudnet2006 - https://phabricator.wikimedia.org/T350479 (10cmooney) p:05Triage→03Medium
[12:43:46] <jinxer-wm>	 (PuppetConstantChange) firing: Puppet performing a change on every puppet run on debmonitor2003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[12:43:47] <jinxer-wm>	 (SystemdUnitFailed) firing: (3) debmonitor-maintenance-gc.service Failed on debmonitor2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:56:17] <jinxer-wm>	 (NodeTextfileStale) firing: Stale textfile for puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[13:56:43] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Put Dell SONiC switches in production - https://phabricator.wikimedia.org/T335028 (10Jclark-ctr)
[14:16:59] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Use default BGP multihop TTL between devices - https://phabricator.wikimedia.org/T350488 (10cmooney) p:05Triage→03Medium
[17:15:24] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Use default BGP multihop TTL between devices - https://phabricator.wikimedia.org/T350488 (10cmooney)
[17:56:17] <jinxer-wm>	 (NodeTextfileStale) firing: Stale textfile for puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[18:17:48] <wikibugs>	 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad: eqiad: Connect IC-374549 - https://phabricator.wikimedia.org/T350504 (10RobH)
[18:17:56] <wikibugs>	 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-eqiad: eqiad: Connect IC-374549 - https://phabricator.wikimedia.org/T350504 (10RobH)
[18:18:27] <wikibugs>	 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10collaboration-services: OTRS/mail: investigate why "T=remote_smtp_signed: all hosts for 'ticket.wikimedia.org' have been failing for a long time" - https://phabricator.wikimedia.org/T297160 (10Dzahn)  ` [mx1001:/var/log/exim4] $ grep -ri "otrs@ticke...
[21:56:17] <jinxer-wm>	 (NodeTextfileStale) firing: Stale textfile for puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[23:51:13] <jinxer-wm>	 (DiskSpace) firing: Disk space puppetmaster1001:9100:/ 5.94% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace