[03:28:27] FIRING: SystemdUnitFailed: netbox_ganeti_ulsfo_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:28:28] FIRING: SystemdUnitFailed: netbox_ganeti_ulsfo_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:28:28] FIRING: SystemdUnitFailed: netbox_ganeti_ulsfo_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:31:01] 10SRE-tools, 06Infrastructure-Foundations: debmonitor: show OS release name in the host view - https://phabricator.wikimedia.org/T240193#10432093 (10hashar) 05Invalid→03Resolved a:03elukey [15:28:28] FIRING: SystemdUnitFailed: netbox_ganeti_ulsfo_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:09:45] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Platform-Team, 10MediaWiki-User-login-and-signup, 07Wikimedia-production-error: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#10432986 (10Krinkle) [16:39:16] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Platform-Team, 10MediaWiki-User-login-and-signup, 07Wikimedia-production-error: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#10433134 (10Krinkle) >>! In T383047#10432915, @Krinkle w... [18:13:55] FIRING: MaxConntrack: Max conntrack at 84.15% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [18:18:56] RESOLVED: MaxConntrack: Max conntrack at 84.15% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:28:28] FIRING: SystemdUnitFailed: netbox_ganeti_ulsfo_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:40:49] 10CAS-SSO, 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE, 07Epic: Improve the JupyterHub services and use CAS/SSO - https://phabricator.wikimedia.org/T260386#10435183 (10VirginiaPoundstone) [21:49:59] 10CAS-SSO, 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE, 07Epic: Improve the JupyterHub services and use CAS/SSO - https://phabricator.wikimedia.org/T260386#10435446 (10Ottomata) [21:55:33] 07Puppet, 10Beta-Cluster-Infrastructure: puppetmaster config in deployment-prep may be inadvertently breaking store,logstash reports? - https://phabricator.wikimedia.org/T218175#10435529 (10bd808) 05Open→03Resolved a:03bd808 At some point in the long life of this bug we moved to Puppet 7 and a new se... [23:28:28] FIRING: SystemdUnitFailed: netbox_ganeti_ulsfo_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:47:55] FIRING: MaxConntrack: Max conntrack at 80.05% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [23:52:55] RESOLVED: MaxConntrack: Max conntrack at 80.05% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack