[01:51:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on thanos-be2008:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[05:52:03] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on thanos-be2008:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[08:16:28] <jynus>	 checking why many snapshots failed today on s4, s3
[08:17:36] <Emperor>	 thanos-be2008 has a disk dying.
[08:18:03] <Emperor>	 still looking for a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1250609 please?
[08:19:31] <jynus>	 the issue is dbprov2007, not db2239, fixing
[08:23:57] <jynus>	 Emperor: checking, but I have a full plate
[08:25:39] <Emperor>	 thanks, should hopefully be an easy review :)
[08:27:16] <jynus>	 what about swift::proxy::memcached_servers mentioned on the docs?
[08:30:37] <jynus>	 Emperor: sent you a question on review
[08:32:37] <jynus>	 ah, ignore me, I got confused by the online editor
[08:37:36] <Emperor>	 Thanks (and sorry for the confusiong) <3
[08:38:09] <jynus>	 enterilly on my side
[08:38:28] <jynus>	 *enterally
[08:38:54] <jynus>	 *entirely
[09:16:14] <Emperor>	 (T419817 opened re the dead disk, alert silenced 'til Monday)
[09:16:14] <stashbot>	 T419817: Disk (sdm) failed in thanos-be2008 - https://phabricator.wikimedia.org/T419817