[06:02:06] !log tools.poty-stuff Updated from 6b633e6 to bdb28be [06:02:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL [07:10:08] !log tools.poty-stuff Updated from bdb28be to 7927bcc [07:10:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL [12:08:42] !log tools.wikiloves Recreate venv in Python3.9 shell (T334932) [12:08:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL [13:35:41] It seems that bridgebot doesn't sync messages between IRC and Telegram :( bd808 [13:50:58] !log tools.bridgebot Restart to hopefully fix broken IRC/Telegram bridging [13:51:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [13:52:17] @lucaswerkmeister I’m away from my laptop, and apparently bridgebot is down. When you get a chance could you check on it please? [13:52:29] yay, looks like a restart worked [13:52:43] oh wow, it even bridged that old message (which was 13:44 UTC on Telegram ftr) [16:26:22] !log tools rebooting root@tools-sgeexec-10-14 due to stuck exec daemon not releasing port 6445 [16:26:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:32:02] !log tools rebooting root@tools-sgeweblight-10-26 due to stuck exec daemon not releasing port 6445 [16:32:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:35:15] !log tools rebooting root@tools-sgeweblight-10-25 due to stuck exec daemon not releasing port 6445 [16:35:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:35:53] !log tools rebooting root@tools-sgeweblight-10-27 due to stuck exec daemon not releasing port 6445 [16:35:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:46:11] !log tools force-rebooting tools-sgeweblight-10-25/26/27 as they got stuck stopping the grid_exec process [16:46:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:27:12] I could use help from someone with rights to edit Horizon hiera for all projects. I broke it out into a separate task to get to the point why, you don't have to read the whole backstory, but you can if you want. https://phabricator.wikimedia.org/T334971 [19:24:56] !log quarry remove db entries ending in semicolon 91d66e53de3b0ab754f89e84f85673212256adab [19:24:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [22:41:59] !log admin resetting rabbitmq on cloudrabbit1003 due to splitbrain [22:42:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL