[02:23:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:23:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:38:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:44:22] <wikibugs>	 07Puppet, 10Cloud-Services, 06SRE, 13Patch-For-Review: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#11049640 (10taavi) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ an...
[11:45:09] <wikibugs>	 07Puppet, 10Cloud-VPS, 06SRE, 13Patch-For-Review: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#11049643 (10taavi)
[11:53:01] <wikibugs>	 07Puppet, 10MediaWiki-Vagrant: Vagrant -> mwvagrant alias in role::labs::mediawiki_vagrant is brittle - https://phabricator.wikimedia.org/T195592#11049703 (10taavi)
[11:55:37] <wikibugs>	 10Mail, 06Infrastructure-Foundations: Qualtrics SMTP email failing - https://phabricator.wikimedia.org/T400882#11049728 (10taavi)
[12:13:42] <wikibugs>	 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: Use vlan trunking instead of multiple physical interfaces - https://phabricator.wikimedia.org/T316114#11049843 (10taavi) 05Open→03Resolved I /think/ this is done for cloudvirts and ceph nodes are tracked separately...
[12:26:25] <jinxer-wm>	 FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[12:46:25] <jinxer-wm>	 RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[13:08:22] <wikibugs>	 10Mail, 06Infrastructure-Foundations: Qualtrics SMTP email failing - https://phabricator.wikimedia.org/T400882#11050088 (10jhathaway) a:03jhathaway
[13:46:05] <wikibugs>	 10Mail, 06Infrastructure-Foundations: Qualtrics SMTP email failing - https://phabricator.wikimedia.org/T400882#11050218 (10jhathaway) @TAndic I ran an initial test and our setup appears to be working correctly. Could we perhaps jump on a call and review the configuration together and send some test emails?
[15:14:47] <wikibugs>	 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, and 2 others: Replace Exim on VRTS servers with Postfix - https://phabricator.wikimedia.org/T378028#11050555 (10jhathaway)
[15:35:30] <jhathaway>	 cdanis: if you have a moment, can you sanity check this patch, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1174741
[15:36:31] <cdanis>	 sure :) I'll take a peek although this is well out of my comfort zone 😅
[15:38:48] <jhathaway>	 yeah, I was just going to merge it in
[15:38:49] <cdanis>	 uh
[15:39:04] <cdanis>	 ah, gerrit is being restarted
[15:39:05] <cdanis>	 I was about to +1
[15:39:20] <jhathaway>	 perfect timing
[15:42:53] <cdanis>	 +1'd
[15:49:05] <jhathaway>	 thanks
[16:05:45] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 13Patch-For-Review: Qualtrics SMTP email failing - https://phabricator.wikimedia.org/T400882#11051027 (10jhathaway) 05Open→03Resolved It appears there were two issues:  #. Our mail server hostname changed from mx1001.wikimedia.org to mx-out1001.wikimedia.org, whi...
[21:03:49] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Emails from wikimediats.zendesk.com fails DMARC policy - https://phabricator.wikimedia.org/T378285#11052031 (10JAbrams) Hi all, The SMTP connector has been successfully configured for all support addresses in Zendesk. Thanks to @jhathaway , the implem...