[08:37:31] !log tools.toolschecker Removing toolschecker cron stuck job to allow retriggering [08:37:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolschecker/SAL [08:59:23] hi [09:01:19] hi [09:24:36] !log admin Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus [09:24:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:24:48] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [09:24:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:26:15] !log admin Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus [09:26:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:26:19] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [09:26:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:29:47] !log admin Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus [09:29:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:29:51] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [09:29:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:32:43] !log admin Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [09:32:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:32:47] !log admin Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [09:32:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:35:34] !log admin Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [09:35:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:35:42] !log admin Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [09:35:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:38:41] !log admin Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [09:38:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:39:01] !log admin Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus [09:39:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:39:13] !log admin Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus [10:39:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:39:26] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:39:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:42:23] !log admin Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:42:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:42:27] !log admin Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:42:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:45:25] !log admin Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:45:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:45:30] !log admin Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:45:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:48:28] !log admin Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:48:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:48:47] !log admin Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus [10:48:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:49:06] !log admin Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus [10:49:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:49:21] !log admin Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:49:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:52:00] !log admin Finished rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:52:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:52:06] !log admin Rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:52:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:55:38] !log admin Finished rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:55:42] !log admin Rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:55:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:55:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:58:17] !log admin Finished rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [10:58:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:58:35] !log admin Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] (T281248) - cookbook ran by dcaro@vulcanus [10:58:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:50:04] !log tools.sge-jobs deploying https://phabricator.wikimedia.org/R2080:072ca798be2c8e95dfe6b86d8b86e28872155e67 [12:50:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sge-jobs/SAL [15:08:48] !log admin-monitoring stopping the nova-fullstack service to allow troubleshooting the current VMs [15:08:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL [16:11:06] !log admin-monitoring cleaning up the nova-fullstack VMs after merging puppet fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/699237 [16:11:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL [17:38:12] !log tools clear error state from tools-sgeexec-0907, task@tools-sgeexec-0939 [17:38:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:22:21] !log tools.lexeme-forms deployed 1f94df1209 (l10n updates) [20:22:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [20:28:33] bd808: looks like the easy fix for T284144 didn't work. I found the source of phabricator module, maybe I can try to run the commands myself to see if I get an error message from git? [20:28:34] T284144: Mirroring tool-spacemedia Diffusion repository to GitHub seems to be broken - https://phabricator.wikimedia.org/T284144