[00:50:06] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [03:33:33] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [04:50:06] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [06:58:05] 10CAS-SSO, 06collaboration-services, 06Infrastructure-Foundations, 10GitLab (Auth & Access), 10Release-Engineering-Team (Radar): Add GitLab to offboarding workflow - https://phabricator.wikimedia.org/T339843#10987189 (10MoritzMuehlenhoff) >>! In T339843#10986537, @Dzahn wrote: > Since infrastructure-secu... [07:02:25] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:33:33] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [08:16:15] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10987300 (10SLyngshede-WMF) Backup created: ` $ sudo systemctl start postgres-dump $ sudo ls -lah /srv/postgres-backup/ -rw-r--r-- 1 postgres postgres 84M Jul 9 07:37 psql-all-dbs-2025-07-09-... [08:17:45] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10987301 (10SLyngshede-WMF) Running netbox-extra update: ` sudo cookbook sre.netbox.update-extras --reason 'T397300 - Netbox 4.0.11 upgrade' -a netbox ` [08:50:06] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:01:06] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10987408 (10SLyngshede-WMF) Checks / cookbooks: ` sudo cookbook --dry-run sre.dns.netbox -t T397300 "Testing Netbox 4.0.11" sudo cookbook --dry-run sre.puppet.sync-netbox-hiera "Testing Netbox... [09:01:11] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10987409 (10SLyngshede-WMF) 05Open→03Resolved [09:14:51] FIRING: [7x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:19:51] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:43:07] 10SRE-tools, 06Infrastructure-Foundations, 10SRE Observability (FY2025/2026-Q1): More frequent Puppet runs on the alert hosts? - https://phabricator.wikimedia.org/T398444#10987493 (10Volans) I wonder if the prometheus servers have a similar behavior of applying changes from puppet exported resources. FYI th... [10:16:08] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10987549 (10cmooney) @ayounsi just a data-point but the QFX5120 in the codfw expansion cage, on 23.4R2.13, are exporting the BFD... [10:53:17] 10SRE-tools, 06Data-Platform-SRE, 10Spicerack: Proposal: adding a kafka admin client to spicerack - https://phabricator.wikimedia.org/T399069#10987669 (10brouberol) [11:02:40] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:07:04] 10SRE-tools, 06Data-Platform-SRE, 06Infrastructure-Foundations, 10Spicerack: Proposal: adding a kafka admin client to spicerack - https://phabricator.wikimedia.org/T399069#10987690 (10brouberol) To illustrate the proposal, this is one of many things you can do with an admin client: `lang=python >>> from ka... [11:33:34] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [12:12:25] FIRING: [2x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:02:25] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:19:51] FIRING: [6x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange