[04:18:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be1074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[05:21:23] <marostegui>	 federico3: https://phabricator.wikimedia.org/T440260
[07:04:50] <slyngs>	 marostegui: Can I unlink https://phabricator.wikimedia.org/T439063 and https://phabricator.wikimedia.org/T439214 from the september switchover, so they are just linked to the next attempt? Or do you need both for clarity?
[07:05:29] <marostegui>	 they can be linked to the next switchover yeah
[07:05:31] <marostegui>	 no problem at all
[07:07:46] <slyngs>	 Thanks, I'm trying to get everything a little more organized to our "Act 2", but it has to work for everyone, not just my weird definition of what orderly means.
[08:18:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be1074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[08:27:20] <marostegui>	 federico3: let me know when you've fixed the depool cookbook as I need to depool pc1 (no rush)
[08:27:48] <federico3>	 yep, I'm chasing it 
[08:28:53] <marostegui>	 <3
[08:44:19] <cezmunsta>	 The Puppet failure on the Swift box looks like an XFS/disk issue - dmesg suggests that it started ~ "[Fri Sep 25 08:57:37 2026]" -  anybody ever touched Swift?
[08:45:37] <marostegui>	 elukey: ^ I recall you've worked with Empero.r on swift disks stuff? I may be wrong ofc : )
[08:51:38] <elukey>	 marostegui: o/ I don't know much sorry, but I can check later if you want
[08:52:07] <marostegui>	 elukey: nah sorry, don't worry, I don't know why I thought you've had debugged things with him
[08:52:20] <elukey>	 probably for supermicro stuff :)
[08:53:07] <marostegui>	 ah maybe that!
[08:53:11] <marostegui>	 thanks
[09:11:19] <cezmunsta>	 https://phabricator.wikimedia.org/T440281 for the moment
[09:12:22] <federico3>	 @marostegui ok, try now :)
[09:12:35] <marostegui>	 ok let me try a depool
[09:13:10] <marostegui>	 federico3: depooled worked
[09:13:33] <marostegui>	 federico3: pool worked too
[09:14:28] <federico3>	 ok, I'm going through the rest of the API to spot any missing bit
[09:15:44] <marostegui>	 what was the issue?
[09:25:54] <federico3>	 I missed some entry points in the webapp/api split as code coverage does not show everything
[12:18:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be1074:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[12:19:35] <cezmunsta>	 ^ adding downtime
[12:55:42] <federico3>	 marostegui: when you have a chance to run a host clone can you let me know if the api is fixed?
[13:04:28] <marostegui>	 The pool and depool worked
[13:04:44] <marostegui>	 I don't have anything to clone right now but maybe you can test with testbed
[13:05:19] <cezmunsta>	 ... just don't use db1176 as the source :)
[15:01:44] <cezmunsta>	 marostegui urandom I have disabled Puppet on ms-be1074 so that I can give it a noop run once https://gerrit.wikimedia.org/r/c/operations/puppet/+/1351957 is merged - Amir gave it a +1 and given that Puppet is failing, this seems a safe approach
[15:03:20] <marostegui>	 Thanks - I don't have much context/experience operating swift so I am not going to be too helpful unfortunately
[15:06:56] * cezmunsta crosses fingers, toes and eyes for good measure
[15:17:41] <marostegui>	 kavitha_: o/
[15:18:07] <kavitha_>	 Thank you marostegui: for re-inviting me
[15:18:33] <kavitha_>	 Yeah, I am in!
[15:19:09] <marostegui>	 no problem!
[16:19:13] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_ring_manager.service on ms-fe1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed