[09:01:35] <jynus>	 any tip on how to handle the new service monitoring after reimage from the recipe, for databases?
[09:02:06] <marostegui>	 jynus: not sure I get your question
[09:02:46] <jynus>	 "Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal.<locals>.check' raised: Not all services are recovered: db2097"
[09:03:03] <marostegui>	 jynus: ah, it is ok, you can let it expire
[09:03:07] <marostegui>	 it will be just a warning
[09:03:09] <jynus>	 the reimage process waits for all icinga checks to be green
[09:03:17] <jynus>	 but they won't until I start mariadb
[09:03:18] <marostegui>	 yeah, but it will timeout at somepoint
[09:03:33] <marostegui>	 they won't be green either after starting mysql, as there will be lag
[09:03:40] <jynus>	 yeah that too
[09:03:50] <marostegui>	 But you can let it timeout, the reimage process will end up just fine
[09:03:57] <jynus>	 maybe there was a parameter to ignore that
[09:03:59] <marostegui>	 and will just issue a warning about that, but it will have exit 0 code
[09:04:06] <jynus>	 or you did some scripting after it, or something
[09:04:22] <marostegui>	 No, I just ignore it and let them timeout
[09:04:38] <jynus>	 thanks, that helps- maybe we can give some feature requests to riccardo to accomodate our needs :-)
[09:11:05] <kormat>	 jynus: ahhhhhhh
[09:11:06] <kormat>	 > Oh, I see the source of confusion- as I am upgrading the hosts at the same time as restarting it, they will appear on both the bullseye and buster lists- hopefully that will be solved in the end.
[09:11:08] <kormat>	 💡
[09:11:23] <kormat>	 that explains something that confused me yesterday. thanks for figuring that out!
[09:11:26] <jynus>	 yeah, no problem
[09:11:33] <jynus>	 as long as people are aware of it
[09:12:00] <jynus>	 you just had hardcoded os_version variable on the code :-P
[09:18:15] <jynus>	 heads up now that I have your attention, that I will be reimaging the dbprov hosts- that means 1h in which some db restores will not be possible and you will have to use the ones on the other dc
[09:19:05] <kormat>	 ack
[09:20:01] <jynus>	 I didn't say the time- towards the end of the week
[09:20:22] <Emperor>	 Friday afternoon! :-D
[09:21:38] <jynus>	 @kormat lol at your last update catching the debian installer
[09:22:03] <kormat>	 aw, crap
[09:22:13] <kormat>	 where's the 'revert' button on phab :(
[09:29:18] <Emperor>	 you can poke around in the database...
[09:29:32] * kormat hisses
[09:34:46] <kormat>	 argh. forgot to put in a downtime for s5/codfw, so expect a bunch of alert resolve messages in a bit, sorry.
[11:33:15] <Reedy>	 https://twitter.com/_sysengineer/status/1511844001556619269
[12:15:54] <kormat>	 👍 👍
[12:25:21] <jynus>	 Reedy: https://en.wikipedia.org/wiki/Pretty_Good_Privacy#Criminal_investigation
[12:30:06] <kormat>	 marostegui: are you currently running a schema change against the s2 codfw primary?
[12:53:50] <marostegui>	 nope
[12:54:02] <kormat>	 ok cool. i'll reboot it shortly.
[13:23:06] <Emperor>	 I (used to) have a printout of my GPG revocation certificate
[13:45:40] <marostegui>	 kormat: you doing db2135?
[13:46:20] <kormat>	 marostegui: already did a while ago
[13:46:29] <kormat>	 something wrong?
[13:46:29] <marostegui>	 ah ok, the proxy complained, I will reload it
[13:46:45] <kormat>	 ahh, i see. i did all the misc codfw primaries, fwiw.
[13:46:45] <marostegui>	 fixed
[13:46:56] <marostegui>	 then you probably need to reload all codfw proxies
[13:47:01] <marostegui>	 I did haproxy2004
[13:47:14] <marostegui>	 yeah, 2001, 2002 and 2003 need reload
[13:47:19] <marostegui>	 Just checked icinga
[13:47:37] <kormat>	 `systemctl restart haproxy`?
[13:47:59] <marostegui>	 systemctl reload haproxy 
[13:48:02] <marostegui>	 that should be enough
[13:48:33] <kormat>	 ok cool, done.
[13:49:06] <marostegui>	 recoveries seem to be arriving
[13:52:59] <kormat>	 just added dbproxy.* to the regex of hosts i watch on icinga.