[03:11:04] <jinxer-wm>	 FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[04:11:04] <jinxer-wm>	 RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[06:02:25] <jinxer-wm>	 FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[06:37:25] <jinxer-wm>	 RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[06:41:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: prometheus-pg-replication-lag.service on maps2012:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:46:25] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: prometheus-pg-replication-lag.service on maps2012:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:11:25] <jinxer-wm>	 FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[11:21:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 80% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[11:26:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 80% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[12:56:25] <jinxer-wm>	 RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[14:15:51] <wikibugs>	 10netbox, 06Infrastructure-Foundations, 06SRE: Grant cn=nda some sort of read only access to Netbox - https://phabricator.wikimedia.org/T302870#11189684 (10Novem_Linguae) FYI, there's currently a patch to remove all NDA access from Netbox in T404494
[15:55:30] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 07Epic: DMARC improvements - https://phabricator.wikimedia.org/T404884 (10jhathaway) 03NEW
[15:56:02] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 07Epic: DMARC improvements - https://phabricator.wikimedia.org/T404884#11190439 (10jhathaway) p:05Triage→03Medium
[16:00:52] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11190457 (10RobH)
[16:03:12] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11190471 (10RobH)
[16:10:22] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11190551 (10RobH)
[16:18:07] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 07Epic: Parse DMARC reports and create a dashboard from data - https://phabricator.wikimedia.org/T404888 (10jhathaway) 03NEW
[16:18:16] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 07Epic: Parse DMARC reports and create a dashboard from data - https://phabricator.wikimedia.org/T404888#11190627 (10jhathaway) p:05Triage→03Medium
[16:27:33] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 07Epic: Parse DMARC reports and expose Prometheus metrics - https://phabricator.wikimedia.org/T404891 (10jhathaway) 03NEW
[16:27:35] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 07Epic: Parse DMARC reports and expose Prometheus metrics - https://phabricator.wikimedia.org/T404891#11190686 (10jhathaway) p:05Triage→03Medium
[17:12:25] <jinxer-wm>	 FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[19:02:25] <jinxer-wm>	 RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag