[10:56:41] <mmecor>	 Hello! A few days ago, the website: https://wapa.wmcloud.org/ worked fine. Now it gives a 502 nginx error. https://wdo.wmcloud.org, which is hosted on the same server, works fine. On horizon, one points at the port 80 and another one at port 81 of the same server. I wonder if there has been any change that can be related to this. Can anyone give me a hand? Thank you very much.
[11:58:02] <mmecor>	 Hello! A few days ago, the website: https://wapa.wmcloud.org/ worked fine. Now it gives a 502 nginx error. https://wdo.wmcloud.org, which is hosted on the same server, works fine. On horizon, one points at the port 80 and another one at port 81 of the same server. I wonder if there has been any change that can be related to this. Can anyone give me a hand? Thank you very much.
[12:07:04] <mmecor>	 !help
[12:07:04] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[12:31:14] <dcaro>	 mmecor looking
[12:31:21] <mmecor>	 thank you! :))
[12:31:34] <dcaro>	 which proje8ct is it?
[12:32:36] <mmecor>	 in horizon, the server is wcdo
[12:32:40] <dcaro>	 👍
[12:35:01] <dcaro>	 the server itself is replying with the bad gateway:
[12:35:06] <dcaro>	 https://www.irccloud.com/pastebin/M4xHI8Ps/
[12:35:48] <mmecor>	 that's very weird. i haven't modified the contents of the server in a looong time - 6 months at least
[12:36:59] <mmecor>	 there are actually four subdomains: wdo.wmflabs.org, which is working well. then wdo-dev, wapa, and wapa-dev, are the ones returning this 502.
[12:37:12] <mmecor>	 or wdo.wmcloud.org, which is the same
[12:40:14] <dcaro>	 From the logs:
[12:40:21] <dcaro>	 2023/08/18 10:31:39 [error] 1605725#1605725: *230 connect() to unix:/srv/wapa/src_viz/dash_apps_wapa_dev.sock failed (111: Connection refused) while connecting to upstream, client: 172.16.5.238, server: wapa-dev.wmcloud.org, request: "GET /favicon.ico HTTP/1.1", upstream: "uwsgi://unix:/srv/wapa/src_viz/dash_apps_wapa_dev.sock:", host: "wapa-dev.wmcloud.org", referrer: "https://wapa-dev.wmcloud.org/"
[12:40:40] <dcaro>	 maybe the uwsgi process that was serving that socket is dead?
[12:40:49] <mmecor>	 let me check
[12:41:40] <mmecor>	 seems like it :/
[12:41:47] <mmecor>	 now restarting it
[12:42:08] <dcaro>	 I get stuff now :)
[12:42:15] <mmecor>	 me too :D 
[12:42:19] <mmecor>	 mystery solved
[12:42:21] <mmecor>	 thank you!
[12:42:22] <dcaro>	 \o/
[12:42:53] <mmecor>	 have a good afternoon!
[12:46:59] <dcaro>	 you too :)
[12:50:03] <jem>	 !help It seems there are overload problems in tools-sgebastion-10
[12:50:03] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[12:50:55] <jem>	 And I get "fork: retry: Resource temporarily unavailable" when I try do... anything
[12:51:01] <jem>	 try to do*
[12:51:06] * taavi looks
[12:51:13] <jem>	 Thanks
[12:54:21] <taavi>	 nfs is acting up, again.. I'll reboot it
[12:55:02] <jem>	 Ok
[12:55:51] <taavi>	 !log tools reboot frozen tools-sgebastion-10
[12:55:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:59:46] <jem>	 Working fine now, thanks, taavi
[14:06:59] <Rook>	 !log paws unrar added
[14:07:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[15:48:25] <framawiki>	 Hi, gyi I can't use toolforge-jobs command on bastion: https://phabricator.wikimedia.org/T344507
[15:53:13] <taavi>	 framawiki: try now? I removed an extra line from your ~/.kube/config
[15:53:36] <framawiki>	 \o/ yes, thanks taavi 
[19:34:27] <wm-bb>	 <Yetkin> LiftWing API times out for some edits?
[20:44:15] <bd808>	 @Yetkin: The machine learning team hangs out in #wikimedia-ml on libera.chat if you'd like to ask them questions.