[00:23:48] Starting/restarting containers on the Kubernetes cluster might feel a bit slow for the next few hours/day. The images have all been updated which in turn means that each Kubernetes exec node will need to pull newer layers into its local image cache when starting a pod. [11:17:02] stashbot is down [11:18:10] CloudVPS network is up and running [11:19:07] Telegram is still seeing doubled messages from IRC [11:19:36] I don't see anything special on stashbot logs [11:20:11] Bot might just need to be restarted [11:22:08] !log tools.stashbot `tools.stashbot@tools-sgebastion-11:~$ bin/stashbot.sh restart` bot was disconnected from IRC for no apparent reason [11:22:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [11:22:23] !log tools.bridgebot Double IRC messages to other bridges [11:22:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [14:00:51] !log project-proxy added proxy-api security group to proxy0[34], opening the proxy API to the public for T319312 [14:00:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL [14:00:53] T319312: Open Openstack APIs to the public internet - https://phabricator.wikimedia.org/T319312 [14:35:27] Lucas_WMDE: thanks for being the first responder to bridgebot going goofy yet again [14:35:43] np :) [14:36:21] I think I’m getting close to scripting the whole ssh + become + restart dance :D [14:43:09] I really should try to figure out how to patch the upstream to ignore the irc server connect events when the client is already connected. (T305487) [14:43:10] T305487: Bridgebot freaks out and sends double messages from IRC to Telegram - https://phabricator.wikimedia.org/T305487 [15:10:09] !log tools restart jobs-emailer pod [15:10:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:44:02] Can’t you use sudo? (re @lucaswerkmeister: I think I’m getting close to scripting the whole ssh + become + restart dance :D) [15:44:31] yeah, something like ssh toolforge sudo -u tools.become ./whatever.sh… [15:44:54] (I don’t remember if it’s called manage.sh or something else, I always get it from the bash history :P) [15:46:03] And have a local ./bin/kickstupidtool.sh [16:00:40] Lucas_WMDE: ssh login.toolforge.org become tool do-something works too [16:03:25] ok [17:48:46] danilo: wmopbot is down