[13:52:39] <marostegui>	 Who's doing the VRTS migration later today? I've got some questions
[14:15:47] <claime>	 topranks: /32
[14:15:51] <claime>	 eesh sorry
[14:16:07] <topranks>	 np.  /128 
[14:16:14] <marostegui>	 XDD
[15:27:34] <inflatador>	  /31 ?
[15:37:31] <eoghan>	 marostegui: arnoldokoth will be leading it, and I'll be around as well. 
[15:38:02] <marostegui>	 eoghan: ok will you guys set vrts in RO?
[15:40:17] <eoghan>	 The mysql replica, you mean?
[15:40:30] <marostegui>	 well in general, how will the schema migration be done
[15:44:26] <arnoldokoth>	 marostegui: Usually we just enable maintenance mode on the VRTS dashboard which prevents all users (except admins) from logging in and terminates all active sessions. 
[15:45:12] <marostegui>	 arnoldokoth: right, that works. If we are going to stop the temporary host I set up, can you please let me know once that is setup and before you do any schema changes?
[15:45:37] <marostegui>	 arnoldokoth: It's scheduled for 18:00 UTC right?
[15:52:08] <eoghan>	 That's correct
[15:52:17] <marostegui>	 ok thanks
[15:52:29] <eoghan>	 marostegui: Would you prefer to leave instructions and one of us could take care of it?
[15:52:37] <eoghan>	 It's late your time
[15:52:56] <marostegui>	 eoghan: sure, you just need to log into cumin1002 and run: sudo db-mysql db1134 -e "stop slave"
[15:53:13] <eoghan>	 Got it, we'll do that between maintenance mode and upgrades. 
[15:53:25] <marostegui>	 and then run sudo db-mysql db1134 -e "show slave status\G" | grep Seconds
[15:53:30] <marostegui>	 and that should show: NULL
[15:53:58] <eoghan>	 Awesome
[15:54:07] <marostegui>	 root@cumin1002:~# sudo db-mysql db1134 -e "stop slave" ; sudo db-mysql db1134 -e "show slave status\G" | grep Seconds
[15:54:07] <marostegui>	          Seconds_Behind_Master: NULL
[15:54:27] <marostegui>	 You could run all in a single command but meh
[15:54:49] <eoghan>	 bytes are cheap, brain power is not (: 
[15:55:49] <eoghan>	 And just to check, we can leave the slave stopped for the time being, right? We're not going to disrupt another service by leaving it stopped overnight for example
[15:55:52] <marostegui>	 topranks: starting the migration now right?
[15:56:00] <marostegui>	 eoghan: yeah you can leave it for as long as you want
[15:56:21] <topranks>	 marostegui: yep planning to in a few mins 
[15:56:30] <topranks>	 but if you need some time that's fine we can hold off 
[15:56:39] <eoghan>	 marostegui: Stellar, thanks for the help. 
[15:56:39] <marostegui>	 eoghan: it also has notifications disabled 
[15:56:43] <marostegui>	 topranks: nope, all done from my side
[15:56:50] <topranks>	 great, thanks!
[15:58:42] <arnoldokoth>	 marostegui: Thanks. (apologies for my disappearance. I had to carry some stuff up the stairs).
[15:58:55] <marostegui>	 arnoldokoth: no worries
[16:24:44] <topranks>	 folks we have the servers in codfw rack b5 moved, ports up on all apart from db2188, we are currently working on that (probably a bad cable or transceiver) 
[16:25:17] <topranks>	 all hosts are pinging apart from that, and for some reason the two ganeti hosts - I will look at the latter while dc-ops are working on db2188 link 
[16:26:27] <topranks>	 db2188 back now after SFP swap 
[16:27:55] <claime>	 topranks: awesome, I saw in the dcops channel it took around 2 minutes to do the whole rack?
[16:28:21] <topranks>	 claime: yeah something like that, I'd not started my timer before they were all done 
[16:28:25] <claime>	 big ggs to dcops seriously
[16:28:32] <claime>	 y'all are wizards
[16:28:33] <topranks>	 papaul and JennH had it set up well with cables all ready to go for quick swaps 
[16:28:39] <topranks>	 100%
[16:29:52] <sukhe>	 nice work team!
[16:29:58] <hnowlan>	 wow, impressive 
[16:30:12] <claime>	 we can go ahead and uncordon/repool yeah?
[16:32:00] <topranks>	 claime: yes fire away 
[16:32:05] <claime>	 fantastic
[16:32:35] <topranks>	 ganeti hosts are also back on, little logic issue with our netbox migration script caused an issue there with the tagged vlans 
[16:37:50] <taavi>	 I was about to deploy a DNS change but the MOTD of dns1004 says there's some switch maintenance going on - am I good to deploy that or should I hold off?
[16:38:00] <taavi>	 topranks maybe? ^
[16:38:25] <topranks>	 taavi: I think you are good to proceed
[16:38:33] <taavi>	 ok thanks!
[16:38:42] <topranks>	 I'd disabled puppet fleet-wide for our netowrk migration, but it's complete now, I'm about to re-enable it 
[16:38:47] <sukhe>	 topranks: puppet is still disabled
[16:38:51] <sukhe>	 ok :)
[16:38:54] <taavi>	 cool, I'm going ahead then
[16:39:05] <topranks>	 yeah puppet disabled shouldn't affect that 
[16:39:56] <sukhe>	 yeah not for the DNS change but I was just reminding that it's still disabled, in case you got busy with the other stuff
[17:02:48] <taavi>	 the sre.puppet.sync-netbox-hiera cookbook is showing a diff for a bunch hosts moving from codfw rack A1 to other racks in the same row, is that expected?
[17:03:34] <volans>	 taavi: I think so, topranks to confirm ^^^
[17:03:58] <topranks>	 taavi: my bad should have thought of that and run it msyelf
[17:04:14] <topranks>	 actually that's not realted to us today - we were in B5 
[17:04:29] <topranks>	 but papaul moved a bunch of things out of A1 yesterday - so yes it is expected 
[17:04:33] <taavi>	 this is https://phabricator.wikimedia.org/T355437 it seems, I merged it
[17:04:49] <topranks>	 yep that's it - thanks!