[00:04:39] FIRING: [2x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:49:39] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [04:04:39] FIRING: [2x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:49:39] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:39:39] FIRING: [8x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:49:39] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:57:06] 10netops, 06Infrastructure-Foundations, 06SRE: Create cookbook to add BGP peering for host by triggering Homer run on correct device - https://phabricator.wikimedia.org/T429488#12032936 (10cmooney) >>! In T429488#12029612, @ayounsi wrote: > Not sure if it's worth adding something temporary to setup BGP on th... [11:39:39] FIRING: [8x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:49:39] FIRING: [8x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:16:01] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06tools-infrastructure-team, 06cloud-services-team (FY2025/2026-Q3-Q4): Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12033583 (10fgiunchedi) >>! In T429451#12030754, @cmooney wrote: > @fg... [13:32:13] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06tools-infrastructure-team, 06cloud-services-team (FY2025/2026-Q3-Q4): Establish a blackbox network probe vantage point into cloud realm - https://phabricator.wikimedia.org/T429451#12033677 (10cmooney) >>! In T429451#12033583, @fgiunchedi wrote: > I'm... [13:49:39] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:02:48] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Terminal configuration for cookbooks - https://phabricator.wikimedia.org/T429129#12033892 (10MoritzMuehlenhoff) >>! In T429129#12030649, @MoritzMuehlenhoff wrote: > Thanks! I'll retest the cookbook tomorrow. That didn't work, puppet-merge leaves some "Intern... [14:30:05] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Terminal configuration for cookbooks - https://phabricator.wikimedia.org/T429129#12034139 (10jhathaway) I think that makes sense, "Internal error!" is the helpful message that the args to the script were wrong. However, the whole dual python and shell script... [15:07:37] 10netbox, 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: netbox accounting report error - - https://phabricator.wikimedia.org/T428936#12034372 (10elukey) @RobH it should be fixed now, see https://netbox.wikimedia.org/extras/scripts/results/356792/ [15:09:51] 10netbox, 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: netbox accounting report error - - https://phabricator.wikimedia.org/T428936#12034379 (10elukey) 05Open→03Resolved a:03elukey [15:50:02] FIRING: [5x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:49:39] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:54:39] FIRING: [5x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:49:39] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:54:39] FIRING: [5x] SystemdUnitFailed: cowbuilder_update_bookworm-amd64.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed