[02:29:42] <jinxer-wm>	 FIRING: [8x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 0.6583% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[06:29:42] <jinxer-wm>	 FIRING: [8x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 0.6583% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[07:46:49] <marostegui>	 dhinus: clouddb1026 has now the same weight as 1013, if you notice/hear something off, let me know
[07:46:55] <marostegui>	 I am doing some checks on my side
[08:20:45] <moritzm>	 dbproxy1028 is down, is that a known issue?
[08:20:55] <marostegui>	 nope, checking
[08:20:59] <moritzm>	 thx
[08:21:06] <marostegui>	 moritzm: did it just go down?
[08:21:15] <moritzm>	 let me check
[08:21:19] <marostegui>	 5 days ago
[08:21:20] <marostegui>	 great
[08:21:25] <moritzm>	 only just noticed if when a software deploy failed
[08:21:39] <moritzm>	 yeah, indeed: 5d 19h 13m 42s
[08:21:52] <marostegui>	 I wonder why it is not showing up on icinga
[08:22:08] <marostegui>	 and why they don't have the ping down alert anymore??
[08:22:12] <marostegui>	 we need to investigate
[08:22:21] <marostegui>	 thanks moritzm I will create a task and take it from here
[08:23:20] <moritzm>	 thx
[08:24:27] <dhinus>	 marostegui: ack, thanks!
[09:33:41] <claime>	 marostegui: if you want to drop the database for apiportalwiki in prod go ahead
[09:34:52] <marostegui>	 claime: starting strong on a monday eh
[09:35:06] <claime>	 I mean you can do it tomorrow if you want
[09:35:08] <claime>	 :P
[09:35:10] <marostegui>	 hahaha 
[09:35:12] <marostegui>	 doing it now
[09:35:14] <claime>	 <3
[09:35:17] <marostegui>	 will log on that task
[09:35:26] <claime>	 Cool beans
[10:29:42] <jinxer-wm>	 FIRING: [8x] DiskSpace: Disk space backup1004:9100:/srv/objectstorage 0.6583% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space  - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[10:40:21] <marostegui>	 jynus: ^ is that something you are taking a look at?
[10:41:11] <jynus>	 I will
[10:41:16] <marostegui>	 thanks
[10:41:51] <marostegui>	 jynus: do you need me to create a task?
[10:42:13] <jynus>	 let me give you some context in case it happens again: 
[10:42:22] <jynus>	 those hosts are scheduled for decommissioning
[10:42:54] <jynus>	 but they are full, but sometimes downtime expires or size changes slightly after a re boot, etc
[10:43:15] <marostegui>	 I see, so that can be ignored?
[10:43:45] <jynus>	 I downtime them when they go off, but it doesn't help that, despite having notifications disabled on puppet
[10:43:55] <jynus>	 alertmanager keeps sending those
[10:44:03] <marostegui>	 jynus: When will they be decommissioned?
[10:44:24] <jynus>	 maybe it is an excuse to do it today, I didn't see any issues with the new media backup hosts
[10:48:46] <jynus>	 maybe the main actionable, if you want to help here is start a conversation of why do we get alerts from hosts marked as enabled_notifications: false
[10:48:58] <jynus>	 with the alert owner
[10:49:30] <jynus>	 owner, not as the one that acts on it (me) but the one that implemented it (obs, someoen else?)
[10:49:52] <jynus>	 as I would consider this a regression compared to the icinga behaviour