[03:40:16] (SystemdUnitFailed) firing: (2) geoip_update_main.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:40:16] (SystemdUnitFailed) firing: (2) geoip_update_main.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:04:37] XioNoX: yeah that's it, I'll address that [09:20:42] thx [11:40:16] (SystemdUnitFailed) firing: (2) geoip_update_main.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:28:16] 10Mail, 10Infrastructure-Foundations: Email sent from wikipedia UI seems to use nondeliverable sender: 550 Administrative prohibition - https://phabricator.wikimedia.org/T207650 (10jhathaway) a:03jhathaway [15:30:00] 10Mail, 10Infrastructure-Foundations: Troubleshooting Mail Delivery Issues from Coupa - https://phabricator.wikimedia.org/T306472 (10jhathaway) a:03jhathaway [15:30:10] 10Mail, 10Infrastructure-Foundations: Some emails coming from Gerrit are being tagged as suspicious by Gmail - https://phabricator.wikimedia.org/T226884 (10jhathaway) a:03jhathaway [15:30:31] 10SRE-tools, 10Infrastructure-Foundations: Diffscan: host off-infra - https://phabricator.wikimedia.org/T265595 (10joanna_borun) p:05Triage→03Low [15:32:48] 10netbox, 10Infrastructure-Foundations: netbox cannot import name 'cas_configuration' - https://phabricator.wikimedia.org/T309610 (10SLyngshede-WMF) 05Open→03Declined We switching to OIDC authentication, once that is in place this is no longer relevant. [15:40:17] (SystemdUnitFailed) firing: (2) geoip_update_main.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:41:09] 10SRE-tools, 10Infrastructure-Foundations: Introduce Spicerack.kafka module, along with the method to transfer offset state between consumer groups and clusters - https://phabricator.wikimedia.org/T291681 (10joanna_borun) 05Open→03Resolved p:05Triage→03Medium [16:03:54] jhathaway: might be "just" about adding the header in https://github.com/wikimedia/mediawiki/blob/fe793c21bb611ba33e222a41b80c3541bd0a666e/includes/mail/EmailNotification.php#L550C4-L550C11 [16:05:46] 10Mail, 10Infrastructure-Foundations, 10SRE: Postfix MTA Profile - https://phabricator.wikimedia.org/T325398 (10jhathaway) p:05Triage→03Low [16:05:57] 10Mail, 10Infrastructure-Foundations, 10SRE: Provision mta-inbound-infra - https://phabricator.wikimedia.org/T325401 (10jhathaway) p:05Triage→03Low [16:06:03] 10Mail, 10Infrastructure-Foundations, 10SRE: Provision mta-outbound-infra - https://phabricator.wikimedia.org/T325402 (10jhathaway) p:05Triage→03Low [16:06:14] 10Mail, 10Infrastructure-Foundations, 10SRE: Provision mta-outbound-wiki - https://phabricator.wikimedia.org/T325407 (10jhathaway) p:05Triage→03Low [16:06:22] 10Mail, 10Infrastructure-Foundations, 10SRE: Provision mta-inbound-wiki - https://phabricator.wikimedia.org/T325406 (10jhathaway) p:05Triage→03Low [16:06:29] 10Mail, 10Infrastructure-Foundations, 10SRE: Replace Null client configs - https://phabricator.wikimedia.org/T325408 (10jhathaway) p:05Triage→03Low [16:06:39] 10Mail, 10Infrastructure-Foundations, 10SRE: Remove Exim based MTAs - https://phabricator.wikimedia.org/T325409 (10jhathaway) p:05Triage→03Low [16:16:18] thanks XioNoX, I'll take a look [16:17:50] 10netbox, 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10bacula: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals - https://phabricator.wikimedia.org/T316655 (10Volans) @jcrespo resuming this from our backlog. Could you please re-chec... [16:20:43] 10SRE-tools, 10Infrastructure-Foundations, 10Wikimedia-Mailing-lists, 10serviceops: Support services VIPs with not marked as VIP in Netbox - https://phabricator.wikimedia.org/T295793 (10Volans) a:03cmooney Assigning to Cathal as per meeting discussion. [16:39:02] 10netbox, 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10bacula: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals - https://phabricator.wikimedia.org/T316655 (10jcrespo) This is the list of backups we keep: ` $ check_bacula.py netbox... [19:40:17] (SystemdUnitFailed) firing: (2) geoip_update_main.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:40:17] (SystemdUnitFailed) firing: (2) geoip_update_main.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed