[01:49:39] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:54:39] FIRING: [5x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:49:39] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:39:39] FIRING: [5x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:55:30] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12036327 (10cmooney) @Papaul FYI for any upgrades next week let's use 23.4R2-S8, older versions have this nasty bug. [07:57:29] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12036329 (10cmooney) Also recording JTAC case number is 2026-0616-761841 [08:18:47] 10Packaging, 10SRE-tools, 06Infrastructure-Foundations: Upgrade prometheus-atlas-exporter - https://phabricator.wikimedia.org/T429672 (10ayounsi) 03NEW p:05Triage→03Low [09:08:14] 10SRE-tools, 06Infrastructure-Foundations: Diffscan: investigate IPv6 support and explore other scanning tooling - https://phabricator.wikimedia.org/T265329#12036538 (10ayounsi) Working on T415347, I'm seeing this in the `diffscan-cloud-infrastructure` instance logs. ` Jun 19 08:57:42 diffscan03 diffscan[86132... [09:21:55] btullis: hello! there is an outstanding diff on the network to push the BGP config for dse-k8s-wdqs2004 is it ok to do it? [09:49:39] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:24:39] RESOLVED: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:28:22] btullis: I've flipped the bgp flag back to false to clear the issue [11:39:39] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:14:44] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:23:02] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Create cookbook to add BGP peering for host by triggering Homer run on correct device - https://phabricator.wikimedia.org/T429488#12036813 (10ayounsi) On the scope creep list we could add a validator to prevent turning on the BGP flag if th... [12:30:14] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06tools-infrastructure-team, 06cloud-services-team (FY2025/2026-Q3-Q4): Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12036831 (10ayounsi) It doesn't help directly for DNS, but blackbox ca... [12:36:41] 10Packaging, 10SRE-tools, 06Infrastructure-Foundations: Upgrade prometheus-atlas-exporter - https://phabricator.wikimedia.org/T429672#12036853 (10ayounsi) [13:02:58] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Traceback with IRC notifications on Trixie - https://phabricator.wikimedia.org/T429681 (10MoritzMuehlenhoff) 03NEW [15:46:13] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Traceback with IRC notifications on Trixie - https://phabricator.wikimedia.org/T429681#12037466 (10elukey) @MoritzMuehlenhoff the spicerack config mentions icinga.wikimedia.org:9200 as host:port combination and they don't seem reachable from cumin2003.