[10:59:10] <taavi>	 !log tools.yifeibot bump quotas per request in T329350
[10:59:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.yifeibot/SAL
[10:59:13] <stashbot>	 T329350: Request increased quota for yifeibot Toolforge tool - https://phabricator.wikimedia.org/T329350
[15:25:38] <isaranto>	 Hi folks o/ I have been having some issues with webservices on toolforge. If I restart a webservice then it cant connect any more and I get 504 erros
[15:26:18] <isaranto>	 here is an example https://wiki-gpt.toolforge.org/
[15:27:16] <isaranto>	 any help appreciated or even telling me to contact someone else  :)
[15:29:38] <taavi>	 let's see
[15:31:10] <taavi>	 when did you restart it?
[15:31:48] <isaranto>	 hmm approx 1.30 hour ago..
[15:32:09] <isaranto>	 if it helps this also happened to the webservice running here https://isaranto-test-ci.toolforge.org/
[15:32:21] <isaranto>	 ~15 minutes ago
[15:36:38] <taavi>	 so far it looks like it's an issue with some of the newer toolforge worker nodes we added earlier today, still trying to find the exact problem
[15:37:38] <isaranto>	 aa so when I do a restart on a kubernetes webservice it spawned a pod in the new nodes?
[15:37:47] <isaranto>	 anyway thanks for looking into this
[15:39:17] <taavi>	 each tool with a webservice runs in a kubernetes 'pod', which is essentially a docker container running on one of the many (~50) worker nodes in the cluster. when a new pod is created it picks a node which is usually the one with the least amount of load at that time
[15:39:28] <taavi>	 when you restart the webservice, it deletes the current pod and creates a new one
[15:41:15] <isaranto>	 ack!
[15:45:55] <taavi>	 !log tools reboot tools-k8s-worker-82 to troubleshoot network issues
[15:45:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:46:44] <arturo>	 taavi: is that the new node?
[15:46:50] <taavi>	 yeah, one of them
[15:47:13] <taavi>	 the pod now re-scheduled to -81, which seems to have the same issue
[15:49:56] <arturo>	 also new?
[15:50:29] <taavi>	 yes
[15:50:46] <arturo>	 I kind of remember worker nodes requiring a reboot before putting into service
[15:51:07] <arturo>	 did you reboot them already? (when you put them into service earlier today)
[15:51:38] <taavi>	 I think the cookbook does that
[15:52:00] <arturo>	 uhm, ok
[15:52:05] <arturo>	 let me know if I can be of any help
[15:55:20] <taavi>	 if you can figure out what's wrong with these nodes, that'd be great
[15:56:19] <arturo>	 ok, you mentioned some network issue
[15:56:34] <arturo>	 is there any specific symptom that you are seeing?
[15:57:27] <taavi>	 yeah. as far as I can tell, the ingress nodes are unable to connect to service ips on the new nodes
[15:57:50] <taavi>	 but a manual curl from `webservice shell` to the service ip of these tools work
[15:59:19] <arturo>	 ok
[16:03:58] <isaranto>	 it works now.. 🎉
[16:36:10] <isaranto>	 thanks for the help folks o/
[16:36:15] <isaranto>	 <3
[16:39:46] <jgleeson>	 hi cloud folks. I just discovered the 'X-Clacks-Overhead: GNU Terry Pratchett' http header and it made smile :)
[16:39:54] <jgleeson>	 me*
[18:30:57] <bd808>	 jgleeson: :) https://gerrit.wikimedia.org/r/c/operations/puppet/+/196893
[18:35:22] <jgleeson>	 love it