[00:23:52] !bash subbu: I'm a rabbit hole factory. when you run out stop by and I'll show you some more. ;) [00:23:52] AntiComposite: Stored quip at https://bash.toolforge.org/quip/tq9ySJQBFFSCpsJzNS7G [00:24:17] hehe, I was thinking about it too 👍 [00:35:45] I keep meaning to look back through my irc logs for the context for https://bash.toolforge.org/quip/AU8FVwpD6snAnmqnLHK5 [01:02:04] * bd808 found the context -- https://phabricator.wikimedia.org/P71889 [06:01:40] That was a typo I made, sorry [06:02:07] (the duplicated message) [12:45:34] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 6e55549156 (l10n updates: lb) [12:45:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [12:45:52] dcaro: ah, okay :D [14:30:51] https://www.irccloud.com/pastebin/P2e3Lpum/ [14:31:10] Does anybody any idea ? [14:35:46] Can you run with ssh -vvv (extra verbose), that might give some more hints [14:36:33] (guessing that you are using ssh cli) [14:47:11] Things I've already checked: he is member of the relevant project (machine-learning) and he has uploaded his SSH key on Horizon (ml project > Instances, Key pairs). I also had him try SSH'ing to the primary bastion (instead of through it) and that didn't work (EPERM) [14:51:00] https://www.irccloud.com/pastebin/56gpf2IK/ [15:03:11] georgekyz: any chance this is a name mismatch? What's your shell name on your laptop and what's your shell name on cloud-vps? [15:03:24] (I ask because you aren't doing ssh @) [15:08:09] the user (from the logs) seems to be `debug1: Authenticating to bastion.wmcloud.org:22 as 'gkyziridis'` [15:09:57] somehow that user is not a member of the `bastion` project [15:11:13] keystone thinks they are [15:11:16] https://www.irccloud.com/pastebin/kVLwdgo2/ [15:12:50] the ssh key in ldap is `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICETf8ll3OpBmT6EaB9/yTk71VwRGPR6GmXW2rUIbQE`, but I don't see it being tried for in the verbose output [15:14:32] yeah, 'groups' says they aren't in project-bastion [15:20:30] georgekyz: it seems like something 'interesting' is happening but it might be an hour or two before I have a change to debug it. Sorry for the delay! I will ping you when I know something. [15:25:14] we're in a meeting for another half hour or so, so no worries. [15:45:37] andrewbogott: Thank you so much! I really do not understand what is happening to be honest. Sorry for creating "interesting" :P No worries and no rush. [16:04:33] georgekyz: the gist of problem is that your Developer account is missing some data in the LDAP directory that should have been added automatically when your account was added to the machine-learning project. [16:27:51] I added myself to a new project and the ldap sync happened as expected. So it's not 100% broken, just broken for George :/ [16:38:21] georgekyz: try now? [19:59:17] ctt-qa-03 VM is up .. yay .. now to configure it. On parsing-qa-02, I see a bunch of users (from CTT) with group wikidev. I don't know if those were automatically created via some puppet code OR if I should manually create those users and groups [20:00:55] users are almost always managed via ldap [20:01:12] so whatever you see on the old host is likely true on the new one (and throughout the project) [20:01:51] Sometimes service users are created by puppet or by debian installers, etc, but it doesn't sound like that's what you're talking about [20:03:51] okay, i guess i will wait for a puppet run and see what it creates. i'll look at docs on wikitech a bit more to see if I can trigger a puppet run manually [20:04:14] 'sudo run-puppet-agent' [20:04:39] But users are managed by ldap and not puppet (except to the extent that puppet sets up ldap.) If you're able to log into the host then ldap is already set up. [20:04:44] What's an example of a user you're wondering about? [20:05:58] on parsing-qa-02 I see 'arlolra', 'cscott', 'ihurbain', 'jgiannelos' .. but none on ctt-qa-03. [20:06:50] let me try puppet-agent again. [20:07:33] my first run failed because i had messed with puppet config on horizon .. there is no testreduce class defined .. i misremembered. [20:07:56] okay, that successfully completed this time, but no other changes to users. [20:08:44] root@ctt-qa-03:~# groups arlolra [20:08:44] arlolra : wikidev project-wikitextexp wmf project-bastion project-deployment-prep project-visualeditor project-tools tools.torproxy tools.suggestor [20:08:53] where are you looking? [20:09:09] ls /home [20:09:36] so i assumed that those would have been automatically created for all the users .. [20:10:22] but, maybe they are initialized on first login? [20:10:38] home dir is created on login [20:10:41] first login [20:10:44] got it. [20:11:16] okay .. let me putter around a bit more then. once i am happy with the basic set up. i'll get to unmounting the cinder volume on parsing-qa-02 and attaching it to ctt-qa-03 and mount it. [20:11:54] ok! [20:12:10] don't forget to resize if (if you want to) before you mount it on the new host [20:13:16] yes .. and looks like some of my setup work is going to be simplified because all the gerrit repos checked out are all on /srv on the mounted cinder volume .. so they will appear as is on the new VM. [20:14:03] mysqldumps + set up and config services in /etc and /lib and we should be good to go. I will discover some new packages to install via apt which I'll record and add it to a script for next time. [20:35:20] got to go afk for a bit .. will continue once I am back online. [21:00:58] !log lucaswerkmeister@tools-bastion-13 tools.lexeme-forms deployed cebad0e4dd (l10n updates: ia, pa) [21:01:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [21:36:47] andrewbogott, okay .. so i unmounted /srv/ on parsing-qa-02 and detached it in horizon. So, now is the resizing step. Can you guide me through that? Is it 'Extend Volume' option in horizon? [21:39:32] looks like it according to https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances#Extend_a_volume [21:46:19] all done following instructions there. [21:47:37] but, I did not run the wmcs-prepare-cinder-volume command because I didn't want it reformatted ... so the last qn. is if I should manually edit /etc/fstab for it. [22:00:46] yes, you should copy the /etc/fstab entry from the old VM to the new one. [22:00:55] subbu: did you do the resize2fs already? [22:03:38] I would also test the new fstab with a reboot [22:06:42] yes, i ran the resize2fs [22:07:12] i am right now repopulating all the databases .. so once that is done, will do the fstab changes and reboot. [22:09:18] great [22:09:54] I'm always happy to hear about a successful drive expansion -- it works reliably but feels like brain surgery [22:09:56] andrewbogott, so parsing-qa-02 is using UUID in fstab and ctt-qa-03 is using PARTUUID .. so if I copy over the fstab entry as is, do i change that UUID for the cinder volume to PARTUUID? [22:10:19] huh... [22:10:22] I'm not sure, try and see :) [22:10:26] (short of googling and trying to understand the differences between them) [22:10:28] ah, ok. :) [22:10:42] maybe that's from the different debian versions [23:11:30] the mysql import is taking forever ... looks like it is close though. i should have purged old dbs and tables before doing this ... but didn't think it would take this long. lesson for next time.