[05:26:36] carane logout/signout piye bro [08:59:51] hello everyone. I receive an email saying "Puppet is having issues on the etytree-a.etytree.eqiad.wmflabs (172.16.0.251) instance in project etytree in Wikimedia Cloud VPS. [09:00:01] etytree is my project but I don't know how to proceed [09:00:12] any help would be really welcome [09:05:50] Ester: did you try running puppet on that machine? [09:07:48] I don't remember the procedure to access it... with ssh? [09:09:37] I remember there was also an interactive webpage where I could see the running instances, but i don't remember what webpage it was [09:23:42] Ester: for access you can check https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances [09:27:00] if I do ssh -J epantaleo@bastion.wmcloud.org epantaleo@etytree-a.etytree.eqiad1.wikimedia.cloud I get Connection closed by UNKNOWN port 65535 [09:27:03] you can also see your running instances and such from https://horizon.wikimedia.org [09:27:42] thanks that's the link i was looking for [09:45:27] there is a fault message "Instance 2aefee74-fdd9-4c87-9f57-9dcc300f55d6 cannot be rescued: Driver Error: [Errno 2] No such file or directory: '/var/lib/nova/instances/2aefee74-fdd9-4c87-9f57-9dcc300f55d6/unrescue.xml' [09:45:27] Code 400" [09:45:49] and status is "error" [09:45:54] what am i supposed to do? [09:49:00] Can you open a task in phabricator.wikimedia.org? I'll give it a look [09:59:30] sure thanks [10:01:45] can you add me as subscriber/assign it to me please? :) [10:02:00] sure [10:04:47] T303922 [10:04:48] T303922: etytree tool fault message - https://phabricator.wikimedia.org/T303922 [10:05:05] hope I did it correcly [10:05:06] cheers [10:06:52] it's ok, I'll try to give it a look today, is it something critical for you? [15:23:53] !log toolsbeta deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a T292238 test to toolsbeta [15:23:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:23:56] T292238: Figure out certificate generation for admission webhooks before we lose the certificates/v1beta1 - https://phabricator.wikimedia.org/T292238 [18:04:07] no dcaro it's not critical. [18:25:45] Hey so I'm trying to run Pywikibot on Toolforge but it errors out whenever I try to create a user object [18:25:58] File "/data/project/shared/pywikibot/stable/pywikibot/page/__init__.py", line 2850, in __init__ [18:25:59]     if title.startswith('#'): [18:25:59] TypeError: startswith first arg must be bytes or a tuple of bytes, not str [18:26:59] Is the problem with the code itself or my configuration [19:55:06] Hello, I got locked out of wmcloud after some failed attempts with my yubikey. Please can anyone help with a reset [19:59:06] damilare: where exactly did you get locked out? what error are you seeing? [20:06:02] I got while trying to ssh into one of fr-tech tunnels. Please see attached image: https://i.imgur.com/isjUXGI.png [20:11:10] what I see on the server side is 'Mar 16 19:20:01 bastion-eqiad1-01 sshd[21530]: Postponed publickey for damilare from (redacted) port 51770 ssh2 [preauth]' [20:11:18] "Postponed publickey" is unfortunately not something I've seen before :/ [20:12:28] That's weird, I ctrl-c'd on the last attempt. Could that be a possible reason? [20:12:53] not sure [20:13:37] one thing I'd try is to swap bastion.wmcloud.org to secondary.bastion.wmcloud.org in your .ssh/config [20:14:12] Ok, doing that now [20:17:06] Got the same error https://i.imgur.com/6AcuOgb.png [20:35:29] damilare: hmmm... I see that you had success at 19:10:21 UTC connecting to bastion-eqiad1-01, and then the "Postponed publickey" events that taavi noted at 19:12:27, 19:13:13, 19:14:59, 19:16:02, and 19:20:01. [20:37:22] damilare: on the target instance (payments), I see an auth failure at 19:10:36 that says is was a failed keyboard-interactive/pam attempt. [20:38:19] I see that you are a project admin per https://openstack-browser.toolforge.org/project/fr-tech-dev and that you only have one ssh public key registered... [20:38:43] * bd808 is running out of ideas of what to check [20:39:32] True I think this may have been the last successful attempt. On the last yubikey attempt, I cancelled it with ctrl-c https://i.imgur.com/C7Gpz36.png [20:42:49] damilare: why would your yubikey be involved here? [20:44:37] tbh I wasn't sure what password it requested there after the ssh key passphrase... So I assumed it was that of the yubikey [20:46:17] I think you got asked for a password because you failed 3 times to unlock your ssh private key and then your ssh client tried to switch to password interactive auth instead. [20:49:37] that's possible, thought it was successful on that last attempt. Please how can we reset this now? [20:57:42] damilare: good question. I can't find anything obvious that is broken. I can see your user and ssh public key from all the involved instances and your account has the rights to access the payments instance. [21:07:37] Probably not helpful but 50% of the time when people can't ssh it's because they forgot to specify @ [21:08:15] oh, nm, if someone already looked at auth.log then it's not that [21:16:38] Thanks @bd808, everyone, maybe I'll wait it out and try again in the morning. If it doesn't work, I'll contact you in order to add a new key [21:17:54] damilare: keys are self serve via https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack and/or https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/ [21:21:41] Ok thanks, this has been really helpful [22:03:05] Hi [22:03:13] wow so many ppl [22:03:45] How to IPASpeaker? [22:04:03] i can't do ipaspeaker [22:09:00] How do I use tools? [22:10:13] im directed by toolforge page [22:10:51] IloveUveryMuch: your questions are not making much sense to me yet. What are you trying to get help with? [22:11:02] How do I use tools? [22:12:46] I am directed by wikitech full stop wikimedia [22:13:25] to this IRC channel [22:13:37] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_introduction [22:14:13] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_introduction#Communication_and_support [22:16:57] https://ipa-speaker.toolforge.org/ [22:19:11] IloveUveryMuch: the user that created that tool account does not seem to have actually deployed the tool. There is no application running and no source code on the server that could run. [22:19:43] Ok, I'll build one :) [22:22:31] IloveUveryMuch: there is some information at https://wikitech.wikimedia.org/wiki/Tool:IPASpeaker about what they apparently intended to build. [22:24:06] i already read it, and I frisked the entire toolforge help pages, technical pages, and documentation pages. [22:27:12] This particular tool would be challenging to deploy on Toolforge. It appears to need a binary application called "eSpeak NG" to generate audio files. That application is not installed on any of our servers today.