[02:30:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:33:30] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [06:30:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:33:30] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [08:57:07] 10SRE-tools, 10Spicerack: Increase the default batchsize of puppet.run() - https://phabricator.wikimedia.org/T397687 (10JMeybohm) 03NEW [08:57:21] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Increase the default batch size of puppet.run() - https://phabricator.wikimedia.org/T397687#10941151 (10JMeybohm) [10:30:41] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:33:30] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [12:03:05] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 07IPv6: Some Foundation clusters do not appear to support IPv6 - https://phabricator.wikimedia.org/T271136#10941924 (10Aklapper) a:05MoritzMuehlenhoff→03None @MoritzMuehlenhoff Removing task assignee as this open task has been assigned for more than two... [12:28:25] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: wait_for_optimal() should ignore acked alerts - https://phabricator.wikimedia.org/T319277#10942407 (10Aklapper) a:05SLyngshede-WMF→03None @SLyngshede-WMF: Removing task assignee as this open task has been assigned for more than two years - See the e... [12:28:35] 10netops, 06Infrastructure-Foundations, 06SRE: Store network users in Bitu/LDAP - https://phabricator.wikimedia.org/T335870#10942412 (10Aklapper) a:05SLyngshede-WMF→03None @SLyngshede-WMF: Removing task assignee as this open task has been assigned for more than two years - See the email sent on 2025-05-2... [12:29:27] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mx-in-lists - https://phabricator.wikimedia.org/T325404#10942459 (10Aklapper) a:05jhathaway→03None @jhathaway: Removing task assignee as this open task has been assigned for more than two years - See the email sent on 2025-05-22. Please assign this t... [12:29:33] 10Mail, 06Infrastructure-Foundations, 06SRE: Provision mx-out-lists - https://phabricator.wikimedia.org/T325405#10942454 (10Aklapper) a:05jhathaway→03None @jhathaway: Removing task assignee as this open task has been assigned for more than two years - See the email sent on 2025-05-22. Please assign this... [12:29:49] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim with Postfix on mail servers - https://phabricator.wikimedia.org/T325394#10942461 (10Aklapper) a:05jhathaway→03None @jhathaway: Removing task assignee as this open task has been assigned for more than two years - See the email sent on 2025-05-22.... [12:29:55] 10Mail, 06Infrastructure-Foundations, 06SRE: Replace Exim null client config with a Postfix null client config - https://phabricator.wikimedia.org/T325408#10942468 (10Aklapper) a:05jhathaway→03None @jhathaway: Removing task assignee as this open task has been assigned for more than two years - See the em... [14:30:41] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:32:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on puppetmaster2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:52:48] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:33:30] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [18:17:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [18:30:41] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:42:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [18:53:03] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [19:01:01] I thought that got fixed [19:01:41] oh I see, patch is still pending [19:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [22:30:41] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:53:03] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [23:33:30] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts