[02:39:48] WMCS team: there's a documentation improvement request at https://wikitech.wikimedia.org/wiki/Help_talk:Adding_disk_space_to_Cloud_VPS_instances [12:14:46] hey folks! I'm looking at cloudvps or toolforge for the purposes of running curl and chromium basically to get an eyeball on what third-party sites are giving our IPs, in order to check our deny/allowlist status while debugging zotero/citoid [12:16:58] i note that rule 4.5 disallows proxying through wmcs but allows exceptions. would it be possible to do so, so that I can run curl and chromium on my development machine, and indeed so that i can reconfigure my local (and printf-ified) version of citoid without having to upload it again? at no point would I be configuring my operating system or my regular browsers to talk through it, it'd be a lot of adding `--proxy=` to command lines [12:50:50] zip: that sounds reasonable to me, as long as it's only for development, and small requests (ex. not downloading the linux kernel source through the proxy), I'd recommend using ssh tunnels or similar as that would allow us to also differenciate easily the traffic, and stop if if needed in case anything goes wrong [12:52:14] feel free to open a task with more details if the ssh tunneling is not enough so we can have a better look [13:02:11] I'm sure SSH tunneling will be fine [15:05:28] !log fnegri@tools-bastion-13 tools.sal webservice restart after a user reported the web page was down [15:05:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL [15:38:38] I created a vm in horizon yesterday, but I'm not able to ssh into it. The server accepts my key, then disconnects, pcc-worker1007.puppet-diffs.eqiad1.wikimedia.cloud [15:38:46] debug1: Offering public key: /home/jhathaway/.ssh/id_wmf_cloud ED25519 SHA256:rOs47upYJ9xHTC/Hx7O27OVi8tkz84aOaZxP53/rwMs explicit agent [15:38:48] debug1: Server accepts key: /home/jhathaway/.ssh/id_wmf_cloud ED25519 SHA256:rOs47upYJ9xHTC/Hx7O27OVi8tkz84aOaZxP53/rwMs explicit agent [15:38:54] anything I can do to troubleshoot? [15:45:33] jhathaway: Access denied for user jhathaway by PAM account configuration [preauth]" is what I'm seeing in the auth.log there. That usually means that the account is not actually a project member. [15:46:09] it sure looks like you would be member per https://openstack-browser.toolforge.org/project/puppet-diffs though [15:47:05] thanks bd808, that is strange, since I am a member of the project, and also spun up the vm [15:47:26] is puppet running correctly on that instance?you need the first run to succeed to provision the auth config [15:47:48] jhathaway: puppet is broken on the instance. that's the root cause I think [15:48:02] taavi: good question, I'm not sure, I saw puppet run on the initial provisioning, but I don't know if it ran to completion [15:48:11] "Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key" [15:48:18] womp womp [15:49:31] In /etc/puppet/puppet.conf I see it setup to talk to puppetmaster.cloudinfra.wmflabs.org as I would expect. jhathaway did you happen to recycle a hostname when making this instance? [15:50:00] yes I killed the first one, because dns wasn't propogating for some reason [15:50:06] not that recycling fixed that issue [15:50:25] perhaps better just to kill this issue and bump the number in the hostname [15:50:35] yeah, that would be my recommendation [15:50:41] thanks [15:51:28] the general problem is that things didn't get cleaned up on the puppetmaster when the prior instance was deleted [15:51:49] this is a known possible problem and why re recommend always using a fresh hostname [15:51:55] *we [15:52:09] makes sense, though unfortunate, reusing names is always dangerous in an infrastructure [16:00:53] It works "most" of the time, but there are lots of places that a cleanup could fail that will blow up reused names. [16:59:20] how long does it take dns to propagate on a new vm? I provisioned one an hour or so ago and dns is still not resolving [17:00:00] jhathaway: usually seconds. andrewbogott ^ [17:00:28] nod, thanks bd808, I don't remember having to wait in the past [17:03:14] jhathaway: I just ssh'ed into pcc-worker08.puppet-diffs.eqiad1.wikimedia.cloud with my root key with no problems. Is it some other instance that is giving you DNS grief? [17:04:13] hmm, strange, that is the one [17:04:16] jhathaway@bastion-restricted-eqiad1-3:~$ host pcc-worker1008.puppet-diffs.eqiad1.wikimedia.cloud [17:04:18] Host pcc-worker1008.puppet-diffs.eqiad1.wikimedia.cloud not found: 3(NXDOMAIN) [17:04:28] maybe it is a problem only with that bastion for some reason [17:05:26] jhathaway: "pcc-worker08", not "pcc-worker1008" [17:05:35] :( [17:05:44] damn it, thanks bd808 [17:05:55] it happens :) [17:06:31] I had the benefit of just looking at Horizon data and not knowing what you meant to type as the name when creating the instance ;) [17:07:10] yeah silly mistake [19:09:57] the problem with switching from tools-login.wmflabs.org to login.toolforge.org is that "ssh too" has only 1 auto-completion for me. "ssh log" requires at least going to "ssh login.toolf" to get the right match :P [19:10:39] s/right/a single/ match [19:11:05] * Krinkle would pay extra for tools-login.toolforge.org :P [19:11:32] * Krinkle meanwhile deletes part of his ssh_known_hosts to try and re-inforce it with failed completion. [19:14:06] Krinkle: make an alias for it in your ssh config. "Host toolforge\n HostName login.toolforge.org" [19:17:26] He, that works too :) [20:12:44] I've yet to update my ssh config alias name from "ssh toollabs". I should probably do that at some point [20:28:47] https://github.com/Krinkle/dotfiles/commit/f574a2c054828ce9e1cfbe7272832558b0508726 [20:28:51] :) [20:34:50] huh, does that still apply the User and IdentityFile of the next Host block? [20:50:20] dev.toolforge.org ? (re @wmtelegram_bot: the problem with switching from tools-login.wmflabs.org to login.toolforge.org is that "ssh too" has only 1 auto-compl...) [21:24:08] @MaartenDammers: TIL - that appears to be an alias for the same host [tools-login.wmflabs.org, dev.toolforge.org] > login.tools.wmflabs.org > instance-tools-bastion-12.tools.wmcloud.org.. [21:24:58] the same *old* host, that is. [21:27:42] login.toolforge and bastion.toolforge are aliases for instnance-tools-bastion-13 instead. [21:30:40] although I haven't heard of that name being deprecated, unlike tools-login.wmflabs.org, so maybe having two bastions with this pointing to the secondary one will keep working. It's still mentioned in docs at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Developing_successful_tools#Pick_the_right_development_environment [21:31:14] the cloud-announce mail from 2w ago suggests it's about maintaining *wmflabs.org DNS [21:31:17] Krinkle: dev.toolforge.org is a different instance than login.toolforge.org [21:31:20] I think dev.t.o just has the lower bastion-x number because it got reimaged to newer Debian before login.t.o, I wouldn’t call it old [21:31:36] yeah [21:31:58] the two were mostly to split NFS traffic in the past [21:32:02] (and I believe {login,dev}.toolforge.org are the only two properly supported hostnames ^^)