[01:23:21] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be1088:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[05:24:05] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be1088:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[05:29:02] <_joe_>	 Emperor: yeah so, the package is imported in bookworm and bullseye on apt-staging according to reprepro list bookworm-wikimedia
[05:29:27] <_joe_>	 and also in trixie-wikimedia
[05:29:33] <_joe_>	 so they are in reprepro
[05:29:49] <_joe_>	 somehow, though, it seems that importing the packages on apt1002 then does not work
[05:30:20] <_joe_>	 but tbh, I don't have time for debugging this, the collab team needs to make this system not break every time someone uses it, or redesign it maybe
[05:32:59] <_joe_>	 and yes, somehow packages still has conftool 5.3.0 instead of 6.0.1, even if reprepro shows the new one
[05:34:38] <_joe_>	 Emperor: https://phabricator.wikimedia.org/P84813 might have something to do with it
[05:36:21] <_joe_>	 so yeah I don't know what went on there but I guess apt-staging is completely broken, and I'm done with this nonsense, I'll import packages by hand.
[05:56:04] <_joe_>	 Emperor: conftool is on debian trixie (imported by hand, yes)
[06:03:21] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on ms-be1088:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[06:41:55] <_joe_>	 I suppose that resolution comes from me uploading the package :)
[08:17:04] <Emperor>	 yeah, it was waiting on being able to install python3-conftool
[13:45:14] <elukey>	 _joe_ thanks a lot! Is it ok for me to cut the new spicerack release, install python3-conftool on cumin nodes etc.. at this point?
[13:45:26] <_joe_>	 yes
[13:45:30] <elukey>	 nice :)
[16:16:59] <Emperor>	 I'm in the market for a +1 to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1202192 please? More codfw ms backends being re-added / drained
[16:25:39] <jynus>	 what is this, the marketplace of +1s?
[16:30:36] <Emperor>	 hopefully XD
[16:30:52] <Reedy>	 lots of checking that SRE mouses work
[16:39:46] <jynus>	 Reedy: it is the process
[21:06:00] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db1181 is CRITICAL: 14.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1181&var-port=9104
[21:14:00] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db1181 is CRITICAL: 15.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1181&var-port=9104
[21:24:00] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db1181 is OK: (C)10 ge (W)5 ge 3.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1181&var-port=9104
[21:37:30] <federico3>	 👀