[00:50:06] <jinxer-wm>	 FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[03:33:33] <jinxer-wm>	 FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[04:50:06] <jinxer-wm>	 FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[06:58:05] <wikibugs>	 10CAS-SSO, 06collaboration-services, 06Infrastructure-Foundations, 10GitLab (Auth & Access), 10Release-Engineering-Team (Radar): Add GitLab to offboarding workflow - https://phabricator.wikimedia.org/T339843#10987189 (10MoritzMuehlenhoff) >>! In T339843#10986537, @Dzahn wrote: > Since infrastructure-secu...
[07:02:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:33:33] <jinxer-wm>	 FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[08:16:15] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10987300 (10SLyngshede-WMF) Backup created:  ` $ sudo systemctl start postgres-dump $ sudo ls -lah /srv/postgres-backup/ -rw-r--r-- 1 postgres postgres  84M Jul  9 07:37 psql-all-dbs-2025-07-09-...
[08:17:45] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10987301 (10SLyngshede-WMF) Running netbox-extra update:   ` sudo cookbook sre.netbox.update-extras --reason 'T397300 - Netbox 4.0.11 upgrade' -a netbox `
[08:50:06] <jinxer-wm>	 FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[09:01:06] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10987408 (10SLyngshede-WMF) Checks / cookbooks:   ` sudo cookbook --dry-run sre.dns.netbox -t T397300 "Testing Netbox 4.0.11" sudo cookbook --dry-run sre.puppet.sync-netbox-hiera "Testing Netbox...
[09:01:11] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10987409 (10SLyngshede-WMF) 05Open→03Resolved
[09:14:51] <jinxer-wm>	 FIRING: [7x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[09:19:51] <jinxer-wm>	 FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[09:43:07] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 10SRE Observability (FY2025/2026-Q1): More frequent Puppet runs on the alert hosts? - https://phabricator.wikimedia.org/T398444#10987493 (10Volans) I wonder if the prometheus servers have a similar behavior of applying changes from puppet exported resources.  FYI th...
[10:16:08] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10987549 (10cmooney) @ayounsi just a data-point but the QFX5120 in the codfw expansion cage, on 23.4R2.13, are exporting the BFD...
[10:53:17] <wikibugs>	 10SRE-tools, 06Data-Platform-SRE, 10Spicerack: Proposal: adding a kafka admin client to spicerack - https://phabricator.wikimedia.org/T399069#10987669 (10brouberol)
[11:02:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:07:04] <wikibugs>	 10SRE-tools, 06Data-Platform-SRE, 06Infrastructure-Foundations, 10Spicerack: Proposal: adding a kafka admin client to spicerack - https://phabricator.wikimedia.org/T399069#10987690 (10brouberol) To illustrate the proposal, this is one of many things you can do with an admin client: `lang=python >>> from ka...