[01:10:03] FIRING: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [05:10:03] FIRING: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:00:49] Hi folks - does IF own the exim4 module in puppet? [08:28:36] If so, would you mind looking at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1333127 / T436615 please? And if not, do you know who I should be talking to, please? [08:28:36] T436615: exim4.conf permissions mean xfs_scrub_all_fail.service cannot run - https://phabricator.wikimedia.org/T436615 [08:38:38] best to add Jesse as reviewer, he did the most work on it in the past years [08:39:53] I vaguely remember that the postinst resets some of the permissions on package updates, maybe test on one of the sretest* hosts by fixing the permissions locally and "apt-get install --reinstall $EXIMDEBS" [08:40:40] because whenever we roll out Exim updates, there's alert spam on some roles (the ones with the heavy variant) about permissions until Puppet ran and fixed it [08:40:48] maybe that is sorted it trixie, not sure [08:45:18] Hm, there is more weirdness, but I'll tag him for sure. [08:46:11] [if I "fix" the perms on exim4.conf the systemd unit complains about being unable to make input files, even though it works if I sudo to mail and try running the script by hand. Am updating the task] [08:57:25] Aaah, I think this is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1116595 [08:59:29] always nice to see that someone already fixed it for trixie [08:59:41] well, in two weeks when the new point release comes out [09:00:23] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Alert on packet loss to core sites from POPs - https://phabricator.wikimedia.org/T435592#12274367 (10ayounsi) Should that be abandoned in favor of RPM probes alerting ? As the RPM probes are more accurate, and generic services monitoring ar... [09:01:41] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12274375 (10ayounsi) Nice work !! >>! In T435855#12253094, @cmooney wrote: > The stats for the magru HE circuits are w... [09:10:03] FIRING: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:10:18] anyhow, the joys of systemd sandboxing, let me count them. [12:21:09] 10SRE-tools, 06Infrastructure-Foundations, 10Phabricator, 13Patch-For-Review: offboard-user: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T420324#12275267 (10Aklapper) @MoritzMuehlenhoff Would you know who could review/test/merg... [12:22:44] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517#12275270 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1003 for host netflow1004.eqiad.wmnet with OS trixie [12:52:51] 10netops, 06Infrastructure-Foundations: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517#12275429 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1003 for host netflow1004.eqiad.wmnet with OS trixie completed: - netflow1004 (**PASS**)... [13:10:03] FIRING: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:15:48] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12275541 (10ayounsi) Fastnetmon .service file changed in Trixie, the process doesn't start anmymore: `name=trixie,counterexample netflow1004:~$ sudo systemctl status fastnetmon ● fastnetmon.serv... [13:23:56] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12275566 (10MoritzMuehlenhoff) That's because in Trixie fastnetmon runs under the fastnetmon system user (while as root in bookworm) and our /etc/fastnetmon.conf is only readable by root. Fixing... [13:24:30] 10netops, 06Infrastructure-Foundations, 06SRE: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909#12275570 (10ayounsi) Adding a 3rd netflow host in eqiad shuffled the targets around, it caused the data to be collected for a short while... [13:44:49] RESOLVED: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:59:35] 10netops, 06Infrastructure-Foundations: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517#12275711 (10ayounsi) Trixie instance is live in eqiad, unfortunately we're still missing metrics on some devices. [15:04:23] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12276053 (10MoritzMuehlenhoff) [16:47:14] 07Puppet, 10Beta-Cluster-Infrastructure: Unduplicate beta cluster hiera keys set both in Horizon and in ops/puppet - https://phabricator.wikimedia.org/T277680#12276764 (10taavi) 05Open→03Resolved a:03taavi