[21:17:36] <btullis>	 Hello. Has anytone done anything to the labstore hosts today? I've got NFS blocking from the stat boxes to /mnt/nfs/dumps-labstore1006.wikimedia.org and /mnt/nfs/dumps-labstore1007.wikimedia.org
[21:43:51] <bd808>	 btullis: hmmm... andrewbogott has been doing things to transition off of labstore 1006/1007 to the new clouddumps1001/1002. But I don't know that there have been anything new today.
[21:44:19] <andrewbogott>	 btullis: nothing should be using those hosts anymore, did I miss something in my grepping?
[21:45:14] <btullis>	 Thanks both. It seems like we did miss something. They were still mounted on the stat boxes.
[21:45:21] <btullis>	 https://usercontent.irccloud-cdn.com/file/UAapS3V6/image.png
[21:45:53] <btullis>	 It caused an issue for jupyter users in particular, but it's OK now I think. https://phabricator.wikimedia.org/T319346
[21:45:53] <andrewbogott>	 you can just s/labstore1007/clouddumps1001/ and s/labstore1006/clouddumps1002/
[21:46:32] <andrewbogott>	 oh, this is probably my mistake for not notifying you properly about manually removing the old mounts :/
[21:46:47] <btullis>	 I did a forced lazy unmount of both mount points. The problem was that all four mount points were still mounted and something was trying to traverse them when starting up jupyterhub servers for users.
[21:47:24] <andrewbogott>	 hm, does that mean you need  a new base image for jupyter that doesn't try to mount those?
[21:47:40] <andrewbogott>	 I can try to temporarily revive those boxes if it's jamming things up
[21:47:46] <btullis>	 No worries. It's all good now anyway. I checked that a puppet run didn't put them back into `/etc/fstab` so we're fine.
[21:48:11] <btullis>	 Thanks, but the lazy unmount worked, otherwise I might have taken you up on that. :-)
[21:48:23] <andrewbogott>	 ok great -- sorry for the surprise