[00:01:07] ok! [00:01:44] lucaswerkmeister, thanks for the reminder of tools-static! [07:39:46] aaah, no CORS allowed from tools-static though :) [07:40:09] anyway *stops thinking about it for now* [08:19:23] sup everyone [08:19:41] enjoying ruining my life? [08:21:43] So why are you doing all this [08:23:48] the topic is awesome [16:46:25] Followed the instructions here: https://wikitech.wikimedia.org/wiki/Help:Access_to_Cloud_VPS_instances_with_PuTTY_and_WinSCP but cannot see my files after logging in [16:46:36] WinSCP [16:51:59] on which host are you when looking for the files? the bastion or the VPS behind the bastion? check the other place too? [16:52:13] using bastion.wmflabs.org [16:52:24] Which hosts should I use? [16:54:02] you should use your VPS but you need to connect VIA the bastion to get there [16:54:11] so.. both [16:54:32] but the files and your user should be on the shell of the VPS after you connect [16:55:04] login-buster.wmflabs.org? [16:55:06] the bastion is just a "jump host" that let's you reach your machine that doesnt have a public IP [16:56:01] try bastion.wmcloud.org [16:56:26] login sounds like you might be using toolforge bastions but you want to get to a machine in cloud VPS, that's different [16:57:21] before trying SCP and copying files, try just normal ssh / putty and see if you can login to your machine [16:57:33] once that works, get back to scp [17:01:11] Toolforge stuff and bastion.wmcloud.org are utterly separate things. Toolforge has it's own bastion hosts at login.tooforge.org and dev.toolforge.org (and a legacy login-buster.toolforge.org) [17:01:34] bastion.wmfcloud.org not found? I can connect to login-buster via putty [17:03:02] bastion.wmcloud.org (no "f") , but see my comment that Toolforge doesn't need you to interact with the Cloud VPS bastions because it has its own. [17:04:19] since the whole thing started with "Help:Access_to_CLoud_VPS..." i assumed toolforge is unrelated. which one are you really using, Yetkin? [17:05:18] https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP [17:09:16] usşng Toolforge [17:14:05] then see only the new link provided by bd808 and ignore the other link [17:26:54] cannot connect to bastion.wmcloud.org [17:27:04] hostname correct? [17:29:16] Yetkin: no, if you are using toolforge it's back to: login.tooforge.org and dev.toolforge.org [17:29:29] see screenshots on the link above [17:29:51] https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP#/media/File:PuTTY_showing_login.toolforge.org_login.png [17:45:33] login.toolforge.org does not work [17:46:41] WinSCP says "connection failed" [17:47:29] man, I can connect using outty. I have the problem with WinSCP [17:48:08] used to connect before but lost the correct hostname... [17:48:59] https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP#How_to_set_up_WinSCP_for_direct_access_to_your_Toolforge_account is all I know. Unfortunately I do not have any personal experience with Windows and WinSCP to better explain the expected usage. [17:49:44] How can I find the valid hostnames? [17:50:51] login.toolforge.org is the primary Toolforge bastion. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart#Connect_to_Toolforge_servers_using_SSH [17:53:14] Yetkin, can you ssh directly to login.toolforge.org with putty or no? [17:54:03] make sure you really have the "SFTP" setting in WinSCP, that seems to include your project name. https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP#/media/File:WINSCP_screen_advanced_settings3.png [17:54:24] I am using login-buster with putty [17:55:01] replace "login-buster" with just "login" in both putty and winscp and try again [17:56:02] if that doesnt work in putty.. first debug that part before you worry about winscp and files [17:56:23] Server refused our key. [17:56:52] ok, now it sounds like a different thing to debug altogether. [17:57:49] (wish I had that ticket to link to again, at this point) [17:58:28] users being able to login to old bastion but not to new bastion.. with the same key.. sounds like it's not on their side ? [17:58:38] oh, do the new bastions have stricter ssh key checks [17:58:56] ah, that could explain it [17:59:01] * bd808 tries to see if there are auth.log traces of a problem [18:00:14] "userauth_pubkey: signature algorithm ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]" [18:00:58] The newer sshd doesn't like superyetkin's key [18:01:43] Has there been a recent change? I used to connect using WinSCP until a few days ago [18:02:24] @Yetkin: yes. Yesterday some legacy names for the the toolforge bastions were changed to point to a newer bastion [18:03:32] Specifically login.tools.wmflabs.org now points to a Debian Bookworm server instead of a Debian Buster server [18:04:26] andrewbogott: ^ fun times. The Bookworm sshd config is rejecting older RSA ssh keys [18:04:39] the actual answer is in "ssh-rsa has been deprecated and in fact, disabled by default for security reasons and should be avoided. Use rsa-sha2-256 or rsa-sha2-512 instead. " [18:04:43] it's only with putty keys [18:05:18] Yetkin: if you can, making a key in a more modern format would be good and should be a fix [18:05:53] * andrewbogott surprised at how many people were using that old bastion [18:06:07] * bd808 is not [18:06:16] Yetkin, the newer ssh servers dont like certain old key formats for security reasons [18:07:15] just checked login-buster again and it works with WinSCP. Is there any EOL for this bastion? [18:08:52] it's about the signature algorithm. "ssh-rsa" keys can be signed with SHA-1, SHA-256.. so it's possible that Linux users can still use keys starting with "ssh-rsa" but Putty users cannot.. if the keys were generated with puttygen [18:08:56] afaict [18:09:06] @Yetkin: the EOL for the Buster bastion is "as soon as possible". WMCS folks have been trying to delete it for months. [18:09:55] Yetkin: I think the easiest fix is if you just make a new key with puttygen but use ECDSA instead of RSA and get that updated in toolforge [18:11:00] @bd808 @mutante thanks for your help [18:11:04] https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/UAMLGQ42CVHLRZ5W2CZBJDJFRNSBT4DC/#IXEIFD3ADOJFRQUQ2KCPAE2DNBZ53JLQ in March was the announce of the bastion switches. T314665 is the decom task [18:11:05] T314665: Toolforge: Replace all bastion with grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665 [18:11:41] why grid-less? [18:12:11] Because we removed the job grid in Feburary [18:13:01] I have some PHP scripts that do not work with Kubernetes [18:13:13] you need to fix them [18:15:33] proposed announce email: https://etherpad.wikimedia.org/p/oldbastion [18:17:03] Yetkin, it sounds like you may not be subscribed to the announcement list; you can do that here: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/ [18:17:44] mutante, bd808, is that announce email useful and is it missing anything? [18:18:03] I could also revert the bastion name switch but I think that just delays this confusion... [18:19:30] I definitely think it’s useful to have this information out there where we can other people to [18:21:38] andrewbogott: I made some small edits, like clarify that it's Debian 10 -> 12 and it shouldn't be all ssh-rsa keys, I think only those generated by putty or especially old ssh keygen versions [18:22:11] like.. just that it starts with "ssh-rsa" alone doesn't tell the full story what algo is used to sign it [18:22:13] thank you mutante [18:23:05] did you replace e.g. with f.e.? I've never seen f.e. before although it is objectively better than latin abbreviation :) [18:26:04] eh, I think I did insert f.e. but didn't mean to overwrite existing letters [18:27:26] andrewbogott: you are native speaker, I am not. it seemed normal to me but Wiktonary says you are right that's "rare" and I wasn't aware it's rare:) https://en.wiktionary.org/wiki/f.e. [18:27:44] Hopefully it wins out eventually [18:28:12] heh:) [18:39:16] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 95846f654a (l10n updates: zh-hans) [18:39:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [19:50:42] abbreviations like `i.e.', 'e.g.', and 'f.e.' are in my opinion jargon that we should seek to remove from our technical communications. I feel that longer phrases such as "by which I mean" and "for example" are easier for non-native English speakers to understand. [19:50:59] thanks for sending that email andrewbogott [20:11:33] I have a stuck pod on tools-k8s-worker-nfs-55: cfdw-28705487-7lg5g. It has been stuck for ~2.5 days. `kubectl delete pod cfdw-28705487-7lg5g` shows `pod "cfdw-28705487-7lg5g" deleted` but it just hangs. I initially ran that command ~12 hours ago, but pod is still there. [20:18:09] JJMC89: :( Usually that means some IO lockup that will never clear until the host node is rebooted. I assume it is blocking you somehow? [20:21:07] It was preventing the job for running for about 2 days. Even though the pod wasn't deleted about 12 hours ago, it did allow the job to make a new one, so I'm not blocked now. [20:23:13] JJMC89: do you mind sharing which namespace (or tool) this belongs to? [20:23:24] Namespace: tool-jjmc89-bot [20:24:16] "rpc error: code = DeadlineExceeded desc = context deadline exceeded" -- k8s go boom [20:24:52] so yeah, some process in the container is an IO locked zombie [20:32:57] !log tools Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc [20:32:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:35:07] out of interest – if I wanted to do the same, is https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Drain_and_undrain_a_node roughly the right documentation? [20:35:11] (except s/toolbeta/tools/) [20:36:47] @ucaswerkmeister: that's what I am following, yes. [20:37:11] alright, thanks [20:38:30] as expected everything but JJMC89's stuck job migrated off of the host with the drain cookbook. [20:40:23] !log tools Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected. [20:40:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:42:35] !log tools Uncordoned tools-k8s-worker-nfs-55 following reboot [20:42:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:43:45] thanks for reporting JJMC89. hopefully your new pod will have better luck and not run into some weird NFS lockup [21:16:03] thanks bd808 - looking good [22:57:29] andrewbogott: "the .wmflabs.org domain … is only working by accident and will probably be broken by us without warning" -- that doesn't apply to web proxies created in Horizon, right? [23:00:30] I just rebuilt google-api-proxy for example, and intentionally kept wmflabs.org (just to avoid updating backlinks) [23:00:39] eventually, yes musikanimal, but no not today [23:01:26] okay :) I figured! I thought maybe the email was talking only about the bastion [23:03:04] All *.wmflabs.org hostnames are legacy. Folks on the WMCS team have largely put their existence out of their minds. Things that are not actively maintained tend to fail, often as collateral damage from active maintenance. [23:04:55] Andrew's statement in the mail was I think scoped to hostnames in the Toolforge project. The bastion FQDN in question was deprecated in 2020 when we introduced the toolforge.org domain. Thus very bit rotted today. [23:06:35] 4 years can pass in a blink for a volunteer, but on the other side of things 4 years is kind of forever for an operational team to remember a deprecated hostname.