[05:43:35] <marostegui>	 btullis: did you reboot dbstore1005? staging database isn't up, so we might run into the same issue we had a few months ago
[06:21:52] <moritzm>	 looking at irc logs it seems to have been rebooted by Amir yesterday evening
[06:22:17] <marostegui>	 mmm, the automated script I guess
[06:22:29] <marostegui>	 Amir1: We need to fix that for dbstore then, as it has the staging database
[06:22:31] <marostegui>	 I am going to start it
[06:22:32] <moritzm>	 yeah, it seems to have treated it as a generic DP mysqld
[06:23:11] <marostegui>	 I have started it, Amir1 make sure to address that case in your script
[08:00:04] <XioNoX>	 hello hello! we're going to do a minor Netbox upgrade, please refrain from any changes if possible.
[08:14:29] <moritzm>	 can you drop a note when you're done, please? I have a ganeti makevm cookbook running, will simply have it retry when Netbox is back up
[08:17:43] <XioNoX>	 moritzm: should be good now
[08:18:08] <btullis>	 marostegui: I haven't reboted the dbstore100* servers yet. I'm scheduled to do so in a little over an hour from now.
[08:18:36] <moritzm>	 XioNoX: thanks, indeed worked fine
[08:18:51] <btullis>	 ...but I guess dbstore1005 doesn't need it now, based on the above.
[08:19:15] <moritzm>	 btullis: no need :-) 1003/1005/1007 have all been rebooted
[08:20:07] <moritzm>	 1005 yesterday, 1003 five days ago and 1007 four days ago
[08:20:31] <btullis>	 OK, cool.
[08:22:01] <marostegui>	 btullis: I think Amir1 did all of them. I thought he talked to you
[08:23:36] <btullis>	 I didn't hear anything, but it's no problem. I sent out a maintenance window announcement for this morning, in case they were in use, but nobody seems to have complained anyway. All good.
[09:56:40] <arturo>	 the decommission cookbook failed for me with an import error https://phabricator.wikimedia.org/T336236#8836173 in case someone is interested cc volans 
[09:57:10] <volans>	 XioNoX: ^^
[09:57:38] <volans>	 arturo: thx, looking, might be related to the deploy of the netbox validators to prod
[09:57:44] <volans>	 that happened few minutes ago
[09:57:49] <arturo>	 ok
[09:58:21] <volans>	 checking logs
[10:21:45] <volans>	 arturo: I'm making a fix, I'll ping once fixed so you can re-run the decom cookbook
[10:22:20] <arturo>	 volans: ok thanks
[10:28:40] <volans>	 arturo: should be fixed, could you retry please?
[10:28:48] <arturo>	 volans: ok, doing so now
[10:29:10] <volans>	 thx, sorry for the trouble
[10:29:20] <arturo>	 no problem at all
[10:29:34] <arturo>	 the fact that we have automation is a blessing :-)
[10:30:01] <arturo>	 volans: now getting:
[10:30:03] <arturo>	 [3/4, retrying in 27.00s] Attempt to run 'cookbooks.sre.hosts.decommission.update_netbox' raised: The request failed with code 400 Bad Request: {'__all__': ['Invalid DNS name: must be a valid FQDN']}
[10:30:23] <arturo>	 the previous run flushed the DNS apparently
[10:30:29] <volans>	 rotfl
[10:30:42] <volans>	 ofc, we empty it, and that's not "valid", sorry my bad
[10:32:31] <arturo>	 will hit CTRL+C to stop it in that step to avoid any more potential "half" steps
[10:32:49] <arturo>	 Sleeping for 3 minutes to get netbox caches in sync
[10:32:49] <arturo>	 ^CCtrl+c pressed
[10:32:59] <volans>	 ack
[10:38:51] <volans>	 arturo: sorry, retry now :)
[10:39:03] <arturo>	 ok, running
[10:40:08] <arturo>	 volans: much better this time
[10:40:11] <volans>	 :)