[15:51:54] Hi, I need some help with my mismatch-finder-staging tool on toolforge again, pls. [15:52:22] be more specific please? [15:53:05] webservice start|stop|restart|status works fine, but any other interaction with kubernetes produces a huge amount of stacktrace error messages (goroutine xyz) [15:53:30] for example 'webservice shell' produces these errors [15:54:00] I am trying to start a continuous job in the background as per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_continuous_jobs [15:54:15] Silvan_WMDE: can you past ethe error stack somewhere? [15:54:21] *paste [15:54:26] even 'kubectl get pods' fails :-/ [15:54:38] if you're doing continuous jobs, I'll point you towards T285944 instead [15:54:39] T285944: Toolforge: beta phase for the new jobs framework - https://phabricator.wikimedia.org/T285944 [15:54:41] yes, just a moment, need to scroll up a thousand lines [15:54:45] xd [15:55:17] but the specific issue you're having is documented at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#%22failed_to_create_new_OS_thread%22_from_kubectl and is caused by go liking to use many os threads by default (and our bastions not liking that) [15:55:19] tools.mismatch-finder-staging@tools-sgebastion-07:~$ kubectl get pods [15:55:19] runtime: failed to create new OS thread (have 40 already; errno=11) [15:55:19] runtime: may need to increase max user processes (ulimit -u) [15:55:19] fatal error: newosproc [15:55:58] argh, I should have searched more... [15:56:07] * Silvan_WMDE reads [16:21:36] hmm, [16:21:36] $ GOMAXPROCS=1 kubectl get pods (as suggested in the docs) [16:21:36] still produces the same error [16:21:54] is there anything else I can do? I stopped the webservice already, no luck [16:22:40] fully logging out and then back in usually helps [16:27:26] Silvan_WMDE: are you using screen or tmux? Or otherwise running multiple interactive ssh sessions to the same bastion at the same time? This is typically the trigger for go thread starvation. The quota is per-user not per session so multiple sessions exhaust things [16:27:56] bd808: interesting [16:28:38] I am, as a matter of fact, logged in to a different tool at the same time [16:28:47] let me try without that session... [16:31:51] yes, that's much better now - thanks! no more "go thread starvation" :-) [16:32:08] maybe that tip is worth adding to the Troubleshooting section? [16:51:20] Silvan_WMDE: good idea. I attempted at . It's a wiki so please do add to this if you think it could be made more clear. [18:26:06] Hello -- How to copy a file from local to Toolforge tool's account over scp? [18:28:22] I don’t know how exactly but the take command should be useful to fix the file’s ownership at least [18:30:38] Okay for not to be an XY problem, I want to copy bot's credential file from local. Is there are easy way (like scp)? [18:36:47] If it's small enough then copy the contents and paste it into nano [18:38:49] patsagorn: in theory you can just scp it, the tool home directories are /data/project//, and then use `take ` to change ownership of the file from your personal account to the tool account [18:39:10] although you need to be careful with files with secrets to ensure they don't end up with permissions that allow everyone to read it [18:40:38] majavah: Thank you! [18:51:59] hi! I've noticed that on several haproxy config templates "owned" by cloud the stats socket doesn't include the option "expose-fd listeners" and that's preventing haproxy from being able to reload its configuration without dropping connections [18:56:45] vgutierrez: Most of that was initially written by someone who's no longer at the foundation, so I can't offer a defense of the current implementation :) If you think we should add it, let's add it. [18:57:10] andrewbogott: I've noticed it during my haproxy tests for the cp servers [18:57:42] and I've enabled it there and submitted a CR for the generic haproxy module https://gerrit.wikimedia.org/r/c/operations/puppet/+/737984 [18:58:21] andrewbogott: right now I don't see any downside of being able to migrate sockets and avoid dropping connections :) [18:58:56] vgutierrez: that will apply it globally without me needing to add it to our configs? [18:59:02] nope [18:59:45] https://www.irccloud.com/pastebin/ugjq3hUT/ [19:00:11] it looks like the haproxy.cfg.erb template has been copied in a few modules [19:00:42] Proper module separation, or something :) [19:00:57] yeah, so each module should have its own CR [19:01:27] mixing in the same CR dbproxies and WMCS k8s instances doesn't seem right [19:01:58] anyways.. I just noticed it and wanted to give you a heads up [19:03:50] Thanks! I just wrote https://gerrit.wikimedia.org/r/c/operations/puppet/+/737986 [19:49:44] !log devtools - removing manually added things in Horizon Hiera that were already in the repo, please don't keep adding in web UI, we don't want to repeat the same thing we did in deployment-prep [19:49:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:26:37] I have a question totally unrelatd to "Wikimedia Cloud", but I suspect one of y'all knows a good answer to. What's a really good timezone converter website? [20:26:40] I want one that (in particular) allows me to create a URL for "November 16, 2pm PST" that doesn't have a 500-character encoding of a GUID. [20:33:19] robla: I use this one: https://www.timeanddate.com/worldclock/converter.html?iso=20211116T220000&p1=224 [20:33:30] that would be a link to Nov 16, 2pm [20:33:46] and 224 is San Francicso [20:34:23] Oh, look at that! Thanks, mutante ! I like it! [20:34:32] :) yw [20:35:33] Maybe not so unrelated to the Wikimedia Cloud − do check out https://zonestamp.toolforge.org/ :) (re @wmtelegram_bot: I have a question totally unrelatd to "Wikimedia Cloud", but I suspect one of y'all knows a good answer to. What's a really good timezone converter website?) [20:38:02] I've been holding a weekly video call every Tuesday at 2pm Pacific Time for a couple of hours each afternoon (where it's okay to join for 5 minutes, say "hi" and leave). Please DM if you want details (since I'd love to say "hi" to many of y'all) [21:16:01] * bd808 waves to robla [21:17:07] * robla waves back, squinting, wondering if he recognizes "bd808" :-) [21:17:48] you know. I'm that kid who likes good sandwiches and tacos :) [21:19:26] Hey robla [21:19:51] howdy RhinosF1 - do you like good sandwiches and tacos? [21:20:03] robla: not tacos so much [21:21:05] yeah, they're not for everyone. It depends on how you define "taco". It's a controversial topic in San Francisco. [21:22:56] Heh! [21:25:54] La Taqueria makes really good tacos, but I'm not sure they make the best burrito in San Francisco. Methodology failure by FiveThirtyEight: https://fivethirtyeight.com/features/americas-best-burrito/ [21:34:23] because of course [21:38:39] Forwarded from Csisc1994: Hello everyone, [21:38:50] Forwarded from Csisc1994: We developed a web service in Flask and we would like to host it on Toolforge using Kubernetes. [21:38:59] Forwarded from Csisc1994: However, we are currently fighting with SSH. [21:39:11] What does fighting mean [21:39:57] I generated a Public Key and added it to Toolforge. But, I failed to SSH using login.toolforge.org. (re @wmtelegram_bot: What does fighting mean) [21:40:34] What error do you get? [21:40:45] Has your account been approved? [21:41:19] Permission denied (Public Key, Host Based) (re @wmtelegram_bot: What error do you get?) [21:42:38] Well, my account is quite old. I think that it is approved. In fact, I added the tool to Toolforge. But, I just need to SSH to add source codes and run it. (re @wmtelegram_bot: Has your account been approved?) [21:42:57] are you using the "UNIX shell username" that appears in https://toolsadmin.wikimedia.org/profile/settings/accounts/ ? [21:43:03] Are you sure you are using the right key too [21:43:32] No, I am using PowerShell now. (re @wmtelegram_bot: are you using the "UNIX shell username" that appears in https://toolsadmin.wikimedia.org/profile/settings/accounts/ ?) [21:43:43] That's not what he aske [21:44:04] If you look at that url, it will say something like: UNIX shell username reedy [21:44:22] you need to make sure you're logging in using that username (which may be different to your username on your local machine) [21:44:33] Yes (re @wmtelegram_bot: If you look at that url, it will say something like: UNIX shell username reedy) [21:45:42] My shell username is different from my username. I know this. My username is Csisc. My Shell Username is Csisc1994. [21:47:38] Let me try again and send a screenshot. (re @wmtelegram_bot: you need to make sure you're logging in using that username (which may be different to your username on your local machine)) [21:47:44] So chances are it's related to your ssh config, and/or whether your ssh key is loaded into the right agent [21:48:24] I will see this. (re @wmtelegram_bot: So chances are it's related to your ssh config, and/or whether your ssh key is loaded into the right agent) [21:48:46] I will try and see what will happen. [21:48:55] uid: csisc1994 [21:49:04] ^ don't capitalize the shell user name [21:49:05] it is case sensitive [21:49:09] ^ this [21:50:53] blames mediawiki for this [21:51:56] * AntiComposite blames Windows [21:52:58] switches blame to UseMod [21:55:58] Well done Mutante. This was the problem I have made. Thank you. (re @wmtelegram_bot: it is case sensitive) [21:57:23] I will be here to ask questions if I face other matters. Thank you for supporting me. [22:01:47] glad it helped :) [22:01:53] you're welcome [23:47:29] bd808: so I saw the checkbox "Mark the repository read-only in Gerrit"..for Scholarships and clicked in Gerrit. or should we leave a final message in that repo or something? [23:49:35] oh, yes, you said that on the ticket too and I overlooked it. nevermind, ignore that, reverted for now [23:49:42] mutante: I guess I would agree with whatever the "normal" decomm process for a gerrit repo belonging to an abandoned project is. I just don't know what that process is :) [23:50:02] I did not think you wanted to actually empty it [23:50:18] I thought first it was supposed to just stay as is but readonly forever [23:50:36] Is blanking the repo what is normal? [23:50:38] hmm, ok, I will ask around tomorrow :) [23:50:45] not sure either [23:51:26] gerrit setting is back to normal (active) for now [23:52:57] i'll dig around another time if we have docs from andre or something [23:55:15] This all makes me sad to think about which is probably why I wrote the task and then never actually did any of the steps myself. Scholarships was my first big solo project for the WMF.