[13:22:25] !log admin Draining 'cloudvirt1029.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [13:22:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:23:10] !log admin Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [13:23:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:24:14] ragesoss: I'd like to shut down wikiwho-api for a little while to do some maintenance on its hypervisor. Is there a good time or bad time for that to happen? (It might be down for as long as a couple of hours) [13:43:48] !log admin Drained 'cloudvirt1029.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [13:43:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:54:43] !log admin restart nova-fullstack on cloudcontrol1003 to pick up bastion ip change [13:54:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:17:54] !log admin Draining 'cloudvirt1030.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [14:17:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:18:40] !log admin Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [14:18:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:19:23] !log admin Draining 'cloudvirt1031.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [14:19:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:20:08] !log admin Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [14:20:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:32:54] !log admin Drained 'cloudvirt1030.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [14:32:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:34:24] !log admin Draining 'cloudvirt1032.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [14:34:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:35:09] !log admin Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [14:35:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:44:47] !log admin Drained 'cloudvirt1031.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [14:44:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:57:46] !log admin Draining 'cloudvirt1032.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [14:57:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:00:40] !log admin Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [15:00:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:01:48] !log admin Drained 'cloudvirt1032.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [15:01:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:02:23] andrewbogott: this morning (now) would be fine. if you give me a ping when it's done so I can make sure everything comes back up as expected, that would be appreciated. [15:02:51] it's going to disrupt some users whenever it happens, but I don't think this will be a particularly high load time. [15:50:57] thanks ragesoss, I'll start that shortly [16:36:29] !log admin Draining 'cloudvirt1033.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [16:36:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:36:37] !log admin Draining 'cloudvirt1034.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [16:36:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:37:14] !log admin Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [16:37:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:37:22] !log admin Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [16:37:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:51:05] !log admin Drained 'cloudvirt1033.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [16:51:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:03:11] !log admin Drained 'cloudvirt1034.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [17:03:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:03:43] !log admin Draining 'cloudvirt1035.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [17:03:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:04:06] aren't the cookbooks a blesing? [17:04:29] !log admin Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [17:04:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:17:50] arturo: they're a blessing for people who aren't trying to have a conversation here :) [17:18:48] perhaps some cookbooks are too chatty, or perhaps we should move the whole !log traffic to #-feed [17:20:09] anyway, we already reduced !log chat for some cookbooks, see https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/770888 for example [17:20:35] !log should probably work both places but then we should move cookbook logging to -feed [17:20:36] andrewbogott: Unknown project "should" [17:20:45] heh [17:20:54] lol [17:21:33] :-) [17:22:29] I do think though that it's very useful when something goes wrong, but yes, probably this chat is not the place for that [17:26:40] Making !log do a thing in -feed is mostly just a matter of stashbot configuration although it may take some code refactoring right now too. I can't remember if the special toolforge + cloud vps project magic used in this channel is setup to be done in another channel with just a config setting. [17:35:33] ragesoss: All done, wikiwho-api should be back up and reachable. [17:35:38] bd808: https://github.com/wikimedia/stashbot/blob/ed2c6c51dac446cda6d4f5e0909e0212854295e0/stashbot/sal.py#L106 [17:35:43] thanks! [17:35:49] It's a 1 line switch [17:36:03] Probably can get rid of -labs at same time [17:36:07] That must be old [17:37:08] RhinosF1: only 5 years or so ;) https://gerrit.wikimedia.org/r/c/labs/tools/stashbot/+/356662 [17:38:03] bd808: I mean I can take it as an excuse to avoid paperwork and waste time logging into gerrit [17:38:58] we give away WMCS stickers to folks patching away old mentions to -labs [17:39:28] (disclaimer: not an actual policy by the WMF or the WMCS team) [17:40:09] That's an excuse not to do paperwork [17:40:31] I wonder if we have crossed the point yet where "labs" has been deprecated longer than it was in use? I think we are close if not past that. [17:43:47] @andrewbogott the server-local storage is the non-Cinder storage? [17:49:00] bd808: https://gerrit.wikimedia.org/r/c/labs/tools/stashbot/+/773307 [17:50:09] ragesoss: that's right [17:50:59] great, thanks [17:51:53] bd808: jenkins liked me? [17:54:33] !log admin Draining 'cloudvirt1036.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [17:54:33] !log tools.stashbot Updated to 015e496 [17:54:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:54:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [17:55:17] !log admin Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [17:55:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:58:42] !log tools.stashbot Added config to allow !log messages in #wikimedia-cloud-feed [17:59:49] !log tools.stashbot Added config to allow !log messages in #wikimedia-cloud-feed [17:59:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [18:00:08] andrewbogott: {{done}} [18:01:08] thanks RhinosF1 [18:02:32] andrewbogott: i can send the followup to move cookbooks there [18:02:57] i assume only wmcs branch [18:03:20] yep [18:04:20] doing [18:06:30] andrewbogott: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/773309 [18:13:17] !log admin Drained 'cloudvirt1036.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [18:13:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:18:20] !log admin Draining 'cloudvirt1037.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [18:18:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:19:04] !log admin Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [18:19:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:43:54] !log admin Draining 'cloudvirt1038.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [18:43:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:44:39] !log admin Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [18:44:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:14:50] !log admin Draining 'cloudvirt1039.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [20:14:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:14:57] !log admin Draining 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [20:15:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:15:34] !log admin Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [20:15:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:15:43] !log admin Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [20:15:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:30:30] !log admin Drained 'cloudvirt1039.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [20:30:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:37:16] !log tools.jouncebot Manual restart after seeing the bot drop out of #wikimedia-operations; Turns out the client rejoined before the restart, but no harm done in restarting [20:37:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL [20:44:07] !log tools.stewardbots `./stewardbots/StewardBot/manage.sh restart` timed out, did not auto rejoin [20:44:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [20:47:36] !log tools.stewardbots add samtar (User:TheresNoTime) as maintainer [20:47:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [20:54:42] !log admin Draining 'cloudvirt1041.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [20:54:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:55:29] !log admin Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [20:55:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:04:41] !log admin Draining 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [21:04:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:07:37] !log admin Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [21:07:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:09:11] !log admin Draining 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [21:09:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:12:05] !log admin Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [21:12:09] !log admin Drained 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [21:12:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:12:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:19:08] !log admin Draining 'cloudvirt1042.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [21:19:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:19:53] !log admin Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [21:19:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:54:23] !log admin Drained 'cloudvirt1042.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [21:54:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:06:21] !log admin Draining 'cloudvirt1043.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [22:06:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:06:25] !log admin Draining 'cloudvirt1044.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [22:06:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:07:11] !log admin Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [22:07:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:08:10] !log admin Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [22:08:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:12:03] !log admin Draining 'cloudvirt1045.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [22:12:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:12:50] !log admin Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster [22:12:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:38:55] !log admin Drained 'cloudvirt1044.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [22:38:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:53:55] !log admin Drained 'cloudvirt1045.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster [22:53:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL