[00:15:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: user@499.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:15:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: user@499.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:15:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: user@499.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:12:56] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service mirror1001:443 has failed probes (http_mirrors_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mirror1001:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[11:22:56] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service mirror1001:443 has failed probes (http_mirrors_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mirror1001:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[13:59:41] <jhathaway>	 Thanks moritzm & elukey for fixing the ruby-sys-filesystem issue, I realized it was an issue last night and rolled back facter on the affected hosts, but I didn't consider the cider use case. I should have left you folks some notes, sorry
[14:04:02] <moritzm>	 no worries, all addressed now
[14:05:15] <moritzm>	 one thing I didn't get to today is Taavi's suggestion to also add a Depends: ruby-sys-filesystem to our bullseye facter package, could you take care of that? 
[14:05:42] <moritzm>	 in Debian it's a Recommends: by default, but makes sense to bump it to a Depends for your infra
[14:07:15] <jhathaway>	 yes, will do
[14:08:08] <jhathaway>	 what about our bookworm package, it has the same issue
[14:10:26] <moritzm>	 ah yes, good point. it's been a bit of a lurking issue, possibly no server has actually used the mountpoints fact on bookworm so we never noticed
[14:10:54] <moritzm>	 but the package change is mostly needed to unbreak Puppet runs (and on Bullseye we rebuild the package anyway)
[14:11:21] <moritzm>	 I think for bookworm we can simply install ruby-sys-filesystem via Puppet, we no hos will ever run into that issue to begin with
[14:12:55] <jhathaway>	 yeah, I'll add it to puppet, one weird corner case is the partitions fact, which will display mount info, if ruby-sys-filesytem is available, otherwise it happily returns all the other data with no error. I saw this behavor on the an-worker hosts which use the partitions fact
[14:13:32] <moritzm>	 ah, I had been wondering earlier today when an-worker1076 was the only server where we had ruby-sys-filesystem installed :-)
[14:13:55] <jhathaway>	 I think it was mistake to not make ruby-sys-filesystem a depends
[14:14:24] <moritzm>	 for bookworm we also don't have any concrete use yet, so one option would also be to initially file a bug in Debian
[14:15:03] <moritzm>	 maybe it should actually also be fixed there, then we could simply move to the fixed package latera
[14:15:04] <jhathaway>	 yeah, though I fear it will bite us when someone tries to upgrade their an-worker nodes to bookworm
[14:16:25] <moritzm>	 we could also test of it changes with 4.10 from trixie
[14:16:49] <moritzm>	 given it's pure ruby, it should probably install fine unmodified
[14:16:49] <jhathaway>	 still the same on sid, but haven't checked trixie yet
[14:17:04] <moritzm>	 so, sid and trixie have the same version currently
[14:17:32] <moritzm>	 or we report to https://github.com/puppetlabs/facter/issues? maybe it's an edge case noone actually noticed so far
[14:18:50] <jhathaway>	 good point I'll file a bug on facter and in debian, I'll also add the depends on bullseye and cut a custom bookworm package with the depends
[14:19:04] <jhathaway>	 unless you think the bookworm change is not worth it
[14:20:30] <moritzm>	 no, let's do that
[14:20:51] <moritzm>	 we can still ditch the custom intenral build if a future Debian change get backported to a Bookworm point release
[14:21:37] <jhathaway>	 sounds good
[16:43:54] <wikibugs>	 10Packaging, 06Infrastructure-Foundations, 10Thumbor, 10Wikimedia-SVG-rendering, 07User-notice: Update librsvg to version ≥ 2.54 - https://phabricator.wikimedia.org/T381674 (10JoKalliauer) 03NEW
[16:46:56] <wikibugs>	 10Packaging, 06Infrastructure-Foundations, 10Thumbor, 10Wikimedia-SVG-rendering, 07User-notice: Update librsvg to version ≥ 2.54 - https://phabricator.wikimedia.org/T381674#10387044 (10JoKalliauer)
[16:48:28] <wikibugs>	 10Packaging, 06Infrastructure-Foundations, 10Thumbor, 10Wikimedia-SVG-rendering: Update librsvg to version ≥ 2.54 - https://phabricator.wikimedia.org/T381674#10387052 (10JJMC89)
[16:50:00] <wikibugs>	 10Packaging, 10Thumbor, 10Wikimedia-SVG-rendering: Update librsvg to version ≥ 2.54 - https://phabricator.wikimedia.org/T381674#10387060 (10hnowlan) Thumbor is currently running with a bookworm image (T355020): ` runuser@thumbor-main-76fcf686cf-w2kxb:/srv/service$ cat /etc/debian_version  12.8 runuser@thumbo...