[07:29:36] <mgerlach>	 Reedy: thanks again for your help. this works now : )
[08:26:38] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 8f91a8c4f315dbe38827c0002933589837574d89 releng: Remove tags RelEng team doesn't own
[08:26:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[09:02:09] <Somnifuguist>	 Hi, I'm getting "502 Bad Gateway" with nginx when sending large chunks of text in a Flask form to my toolforge webservice. Smaller chunks work fine. UWSGI.log says "OSError: write error". Is there some config option to fix this? I'm using the standard python 3.7 kubernetes backend. Thanks.
[09:20:07] <Lucas_WMDE>	 Somnifuguist: yes, you can configure it in ~/uwsgi.ini, see e.g. https://phabricator.wikimedia.org/R2469:fcb85ef502036982a3f8c554f18df6c8924c2ffa
[09:46:44] <Somnifuguist>	 Lucas_WMDE Thanks! Do I have to change the "webservice --backend=kubernetes python3.7 start" command to include the "uwsgi.ini" config file? I've copied the example file you linked to ~/ but no luck so far.
[10:07:55] <Lucas_WMDE>	 IIRC just placing that file in the tool home directory should be enough
[10:07:59] <Lucas_WMDE>	 (and then restart the webservice)
[10:09:53] <majavah>	 the correct path is ~/www/python/uwsgi.ini
[10:10:44] <Lucas_WMDE>	 oh, is it?
[10:10:54] <Lucas_WMDE>	 duh, it even says that in the commit I linked 🤦
[10:10:57] <Lucas_WMDE>	 sorry
[10:11:20] <Lucas_WMDE>	 I just remembered that I had to link it somewhere else out of ~/www/python/src/ (which is the git repo root)
[10:11:32] <Lucas_WMDE>	 Somnifuguist: what majavah said
[10:11:47] <majavah>	 i wonder why it is not in ~/www/python/src
[10:12:26] <Lucas_WMDE>	 maybe it should be, now that `webservice` has also started to look for the service.template there
[10:29:12] <Somnifuguist>	 Thanks to both. I think I'm now running into the overarching nginx header size limit. Not sure how a user can send a largish chunk of wikitext to a server for processing otherwise, though.
[10:31:00] <majavah>	 how large chunk is that?
[10:37:22] <Somnifuguist>	 about 12k bytes. Just a small script to reorder tables.
[10:40:19] <majavah>	 huh, that's nowhere near the body size limits on the toolforge proxies
[10:41:53] <majavah>	 which tool?
[10:44:05] <Somnifuguist>	 Probably an error on my side then :)  https://wiki-tennis.toolforge.org/misc/
[10:48:18] <Somnifuguist>	 I tried various "buffer-size"s (50k, 200k, 1m), but no luck. The input field requires a wikitext table to be pasted into it.
[10:50:43] <Lucas_WMDE>	 uwsgi buffer-size doesn’t include the request body (https://uwsgi-docs.readthedocs.io/en/latest/Options.html#buffer-size), so that shouldn’t be the issue
[10:51:30] <Lucas_WMDE>	 apparently there’s also limit-post https://uwsgi-docs.readthedocs.io/en/latest/Options.html#limit-post
[10:51:42] <Lucas_WMDE>	 (from https://stackoverflow.com/a/25932856/1420237)
[10:58:55] <majavah>	 this is what I see on the kubernetes ingress proxy (that directly then connects to the webservice): "[error] 7009#7009: *70482946 upstream sent too big header while reading response header from upstream"
[11:00:31] <majavah>	 so it's trying to send large headers, not a large body? any clues what headers that might be?
[11:04:43] * Lucas_WMDE imagines a request with such a large body that the decimal representation in the Content-Length header overflows
[11:06:44] <Somnifuguist>	 Not sure. The request is "https://wiki-tennis.toolforge.org/outputmisc/?script=reverse&input=[wikitext chunk] " Is that what you're referring to? I'm afraid I'm quite new to the web side of things.
[11:07:20] <Lucas_WMDE>	 oh, it’s a GET request?
[11:07:47] <Lucas_WMDE>	 hm, the form on https://wiki-tennis.toolforge.org/misc/ has method=post…
[11:12:32] <Somnifuguist>	 It should be post the data in the fields, then get the "output" page where the processed text can be copied.
[11:13:20] <majavah>	 i think it's failing because the URL it creates to the output page is too long
[11:15:21] <majavah>	 unless there is a good reason not to, i would just remove the separate output page and just return the result on form submission instead of redirecting to separate page for that
[11:15:45] <Lucas_WMDE>	 +1
[11:20:10] <Somnifuguist>	 That makes sense. I blindly followed the template for handling the requests used by the creator of the other scripts in that tool (who's also inexperienced) due to my own ignorance re the web side of things. Evidently that's not the way to do things. I'll do some learning now. Thanks for both of your help.
[12:17:52] <Somnifuguist>	 Now working as intended. Thanks again.
[12:51:08] <tn>	 oh hihi majavah o/ just replied to you on phab
[15:39:14] <arturo>	 !log toolsbeta deploy toolforge-jobs-framework-api git version d85d93ee1c5d4be6a526cf83e806b2679dde3875
[15:39:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[15:55:46] <arturo>	 !log tools added toolforge-jobs-framework-cli_2_all.deb to buster-{tools,toolsbeta} (T285944)
[15:55:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:55:52] <stashbot>	 T285944: Toolforge: beta phase for the new jobs framework - https://phabricator.wikimedia.org/T285944
[16:12:12] <arturo>	 !log tools deploy toolforge-jobs-framework-api git version d85d93ee1c5d4be6a526cf83e806b2679dde3875 (T285944, T286107, T285979, T286485, T286107)
[16:12:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:12:21] <stashbot>	 T286107: toolforge-jobs: Allow specifying arguments to commands - https://phabricator.wikimedia.org/T286107
[16:12:21] <stashbot>	 T285944: Toolforge: beta phase for the new jobs framework - https://phabricator.wikimedia.org/T285944
[16:12:21] <stashbot>	 T285979: toolforge-jobs with --wait hangs indefinitely if the job fails - https://phabricator.wikimedia.org/T285979
[16:12:22] <stashbot>	 T286485: toolforge-jobs: figure out logging  - https://phabricator.wikimedia.org/T286485
[17:35:40] <lucaswerkmeister>	 for some reason I can’t run `kubectl` in the notwikilambda tool anymore – apparently it “failed to create new OS thread (have 41 already; errno=11)”
[17:35:50] <lucaswerkmeister>	 (on sgebastion-07 – I’ll try dev-buster now…)
[17:36:07] <lucaswerkmeister>	 I don’t think I have any other processes running on the bastion under the tool account, apart from the shell
[17:37:21] <lucaswerkmeister>	 ok dev-buster (aka sgebastion-11) seems to work better
[17:37:48] <bstorm>	 Ahh the threads limit
[17:38:22] <bstorm>	 The tool account AND your personal account and any other tool accounts you are running from there matter
[17:38:31] <lucaswerkmeister>	 huh, ok
[17:38:32] <bstorm>	 cgroups know you are sudoed
[17:38:41] <bstorm>	 So that could be it
[17:38:43] <lucaswerkmeister>	 then I need to double-check
[17:38:51] <lucaswerkmeister>	 but I don’t think I’m running that much bastion stuff from other accounts either…
[17:39:08] <bstorm>	 kubectl made us raise those limits before
[17:39:18] <bstorm>	 It does a lot of threading
[17:39:37] <bstorm>	 golang 🤷🏻‍♀️
[17:39:42] <lucaswerkmeister>	 yeah, I noticed bash said it’s 250 now
[17:39:47] <lucaswerkmeister>	 I remember it being something like 14
[17:40:05] <bstorm>	 We may just have to increase it to account for kubectl upgrades
[17:40:12] <lucaswerkmeister>	 at least that got me into the habit of using `exec become` instead of just `become` ^^
[17:40:18] <bstorm>	 goroutines count against thread limits it seems
[17:40:28] <bstorm>	 cgroups don't buy the hype
[17:40:41] <bstorm>	 :)
[17:41:22] <bstorm>	 If you make a ticket, I can try to revisit that today. I'm leaving early today, so I cannot guarantee, but this seems important.
[17:41:33] <lucaswerkmeister>	 I’ll create one later
[17:41:36] <bstorm>	 thank you!
[17:41:47] <lucaswerkmeister>	 (looking into T286724 right now thanks to majavah ^^)
[17:53:28] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> only mount ~/www/python into pygments-server pod (T286724)
[17:53:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[18:24:14] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> don’t mount service tokens into pods (T286724)
[18:24:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[18:49:56] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> block external traffic to internal pods (T286724)
[18:49:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[20:09:15] <balloons>	 !log admin-monitoring remove stale/errored fullstack tests
[20:09:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL
[21:05:40] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> updated pygments-server to 2d4ab81a68 (prevent open(); T286724)
[21:05:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL