[07:49:02] Hi! Quick question: Is anyone else having issues with SFTP access to Toolforge? Worked like a charm before, but i constantly get "Remote side unexpectedly closed network connection" since Friday immediately when I try to log in with WinSCP. :-/ [09:05:43] tkarcher: I can take a look [09:05:51] which bastion are you using? [09:07:01] Thanks! I'm using login.toolforge.org [09:08:13] SSH via PuTTY still works fine (if this helps narrowing down the issue) [09:20:28] ok, let me check the logs [09:23:29] tkarcher: I only see successful login attempts in the logs (send them to you in PM) [09:23:54] I wonder if WinSCP is well configured to use your user/key [09:24:17] tkarcher: would you like to try WinSCP while I stare at the logs? [09:26:41] Sure. [09:26:53] ok, go ahead! [09:27:00] Done [09:27:25] And again. [09:27:51] https://www.irccloud.com/pastebin/Ia7zFEss/ [09:28:52] I believe the server thinks the client (your side) closed the connection [09:32:27] Hm. Firewall issue maybe? Good to know that Toolforge is up and running, at least. Thanks for the analysis! [09:48:47] Oh! While analysing the WinSCP log, I just stumbled across the following line: ! 2021-06-14 11:43:53.714 sudo: usr/lib/sftp-server: command not found [09:49:38] I configured WinSCP to log in as my tool (as it is recommended in the docs). Is this not working anymore? [09:55:30] mmm [09:56:14] you should use WinSCP with your personal account as described here:https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP [09:56:51] I do! [09:57:01] But then I also do that: "Switch to the tool account by giving the protocol option: [09:57:02] sudo -u tools.PROJECT-NAME /usr/lib/sftp-server" [09:57:11] (Same page, further down) [09:57:58] wait, are you missing the heading `/` symbol? [09:58:33] /usr/bin/sftp-server does not exist on tools-sgebastion-07 [09:58:33] aborrero@tools-sgebastion-07:~$ which /usr/lib/sftp-server [09:58:33] /usr/lib/sftp-server [09:58:52] ah, it's lib [09:59:15] Sh*t! Yes, that was it! But when did I change that?! Strange... [09:59:34] It was the missing slash! [10:00:10] I'm connected now. Thanks again! [10:02:12] 🎉 [10:13:05] !log admin setting ssd to debug mode on tools-sgeexec-0917 (T284130) [10:13:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:13:10] T284130: Sudo sss directives failing intermittently - https://phabricator.wikimedia.org/T284130 [10:19:31] !log toolsbeta deploying toolforge jobs-framework-api in kubernetes (just a test) (T283238) [10:19:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:19:34] T283238: Toolforge: develop jobs-framework-api - https://phabricator.wikimedia.org/T283238 [14:06:41] !log tools.pbbot Deploy 3ad1e97: filter out category members (Wikinews webapp) [14:06:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pbbot/SAL [14:18:46] !log logging raised instance limit to 16 T284662 [14:18:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Logging/SAL [14:18:49] T284662: Request increased quota for logging Cloud VPS project - https://phabricator.wikimedia.org/T284662 [16:33:11] I just used Toolhub to look up a tool for the first time ever [16:34:48] hare, that's awesome! How was the experience? [16:35:03] I looked up the name of the thing and found it and clicked on it and got to the thing I wanted to go to [16:36:25] In other words, massive success [16:37:01] is it hard to overstate your satisfaction? [16:43:17] cc bd808 [16:47:33] yay! [17:41:12] bd808: do you think eastern oregon will succeed in joining idaho? [17:41:34] no [17:42:45] neither will northern idaho, eastern washington, and easter oregon succeed in creating a 51st state :) [17:42:54] and california won't split into 3 states [17:42:58] i feel like leaving a state and joining a bordering one is way easier than creating a new state [17:43:10] thank you, i've been obsessed with this ever since it came up. there is something i find extremely uncanny about a state of idaho that is larger than texas [17:43:31] i am not opposed to it, it just feels very weird [17:44:06] I think upon investigation they will figure out it will cost way too much to implement and not a whole lot in return [17:45:03] its wild eyed anti-government right winger fantasy [18:40:13] !log metricsinfra Add majavah as projectadmin T284938 [18:40:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Metricsinfra/SAL [18:40:16] T284938: metricsinfra project admin for Majavah - https://phabricator.wikimedia.org/T284938 [19:04:42] hey there wmf cloud folks! [19:11:24] I'm reading through the Cloud VPS / toolsforge docs wondering if it would possible to use a cloud VPS instance as a test server reachable via HTTPS. We're integrating with Applepay and to test the integration we need a live endpoint that Apple can send traffic to over HTTPS. Ideally we wanna tunnel into the server/machine to intercept the traffic sent via 443 to the Cloud VPS with an SSH remote [19:11:25] tunnel, this would let multiple people test the integration from the one box, and then process that locally within our development environment. [19:37:53] !log tools.vpsalertmanager add self as maintainer T284938 [19:37:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.vpsalertmanager/SAL [19:37:56] T284938: metricsinfra project admin for Majavah - https://phabricator.wikimedia.org/T284938 [19:43:02] jgleeson: did you just reveal a future apple pay feature? ;) [19:50:55] jgleeson: yes, actually all proxied webserver access to Cloud VPS and Toolforge is TLS secured by default. https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet is the thing you would likely want to use. [19:51:57] the ssh tunnel bits are probably a bit more complex to work out, but ultimately that should be possible [20:05:33] awesome bd808! [20:05:56] does the cloud stack do SSL termination by any chance? [20:06:10] hare: maybe :P [20:06:41] yes. the TLS will be terminated at the front proxy. We do not (yet) have a solution for end-to-end TLS through the front proxy to the backing instance. [20:07:38] Can also do it without the proxy if you need to, but then TLS termination is your problem [20:07:51] that should be fine for us. We just wanna accept traffic I think [20:08:24] The SSH remote thing did sound odd [20:09:57] @Alex it's a convenience hack for developing the MW integration. They want a single TLS endpoint for the testing service that can be redirected to a local dev stack by the appropriate people to simplify the account setup bits on the payment gateway provider side [20:10:12] ^^^ [20:10:30] * bd808 has done these horrible things before :) [20:10:35] :) [21:29:19] !log toolsbeta deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 T282975 [21:29:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [21:29:24] T282975: Create Kubernetes ingress for tools running on the grid engine to remove dynamicproxy - https://phabricator.wikimedia.org/T282975 [21:29:44] !log wm-bot copied freenode config for #cvn-sw to libera [21:29:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wm-bot/SAL [22:21:29] !log tools push docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web:testing to test staged os.execv (and other patches) using toolsbeta toollabs-webservice version 0.75 T282975 [22:21:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:21:33] T282975: Create Kubernetes ingress for tools running on the grid engine to remove dynamicproxy - https://phabricator.wikimedia.org/T282975 [23:56:30] !log tools.lexeme-forms deployed 70efbdc1a7 (update volitive item ID) [23:56:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [23:58:14] !log tools.lexeme-forms deployed 626b73a005 (l10n updates) [23:58:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL