[01:54:52] !log cloudstore emptying and deleting project; as far as I know this was only used by Brooke for NFS testing [01:54:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudstore/SAL [02:03:16] !log commonsarchive shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:03:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commonsarchive/SAL [02:04:41] !log discourse shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:04:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Discourse/SAL [02:05:52] !log extdist shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:05:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Extdist/SAL [02:06:43] !log fastcci shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:06:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Fastcci/SAL [02:08:11] !log getstarted shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:08:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Getstarted/SAL [02:09:22] !log huggle shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:09:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Huggle/SAL [02:10:23] !log mix-n-match shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:10:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mix-n-match/SAL [02:11:16] !log mwstake shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:11:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwstake/SAL [02:12:42] !log osmit shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:12:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Osmit/SAL [02:13:16] !log packagist-mirror shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:13:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Packagist-mirror/SAL [02:14:29] !log rcm shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:14:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [02:15:19] !log rcm (I didn't actually delete anything due to the hosts being upgraded in place) [02:15:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [02:15:57] !log sciencesource shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:15:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Sciencesource/SAL [02:16:57] !log traffic shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:16:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Traffic/SAL [02:17:48] !log wcdo shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:17:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wcdo/SAL [02:18:38] !log wikiapiary shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:18:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikiapiary/SAL [02:19:22] !log wikidata-history-query-service shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:19:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-history-query-service/SAL [02:20:09] !log wikidocumentaries shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:20:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidocumentaries/SAL [02:20:55] !log wikilabels shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:20:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [02:21:39] !log wikitextexp shutting down Stretch VMs as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/DSSVK2ZM3KHJI4HVM53JFNMECFMMCCG3/ [02:21:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikitextexp/SAL [06:54:50] !log cloudinfra rebooting puppetmaster-03 do to disk errors [06:54:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [07:03:31] !log cloudinfra manually running fsck through costole on puppetmaster-03 do to disk errors (T313380) [07:03:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [07:03:34] T313380: PuppetAgentStaleLastRun - cloud-puppetmaster-03 - https://phabricator.wikimedia.org/T313380 [09:57:42] morning, I'm having issues spawning an instance under the traffic project, current error "[Error: Build of instance ba920cfe-28f3-4ebe-acee-5e3a858b9bd5 aborted: Failed to allocate the network(s), not rescheduling.]." [09:58:15] failed instance --> https://horizon.wikimedia.org/project/instances/ba920cfe-28f3-4ebe-acee-5e3a858b9bd5/ [10:32:51] yep, we are having issues since the troubles with the network this morning [10:33:26] thanks for the ping, trying to get it working :/ [10:34:41] thx [10:43:19] !log tools.stewardbots SULWatcher/manage.sh restart [10:43:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [13:17:52] !log admin restarting the whole rabbit cluster (T313400) [13:17:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:18:00] T313400: 2022-07-20 CloudVPS unstability after network outage - https://phabricator.wikimedia.org/T313400 [14:16:17] !log admin stopping rabbin on cloudcontrol1004, leaving only 1003 alive (T313400) [14:16:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:16:21] T313400: 2022-07-20 CloudVPS unstability after network outage - https://phabricator.wikimedia.org/T313400 [15:51:38] !log admin things seem stable now with one rabbit node, trying to bring up a second (T313400) [15:51:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:51:45] T313400: 2022-07-20 CloudVPS unstability after network outage - https://phabricator.wikimedia.org/T313400 [16:26:17] !log admin things seem stable, trying to bring up a third, cloudcontrol1005 (T313400) [16:26:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:26:21] T313400: 2022-07-20 CloudVPS unstability after network outage - https://phabricator.wikimedia.org/T313400 [17:10:40] !log admin things seem stable, trying to bring up a fourth rabbit node, cloudcontrol1006 (T313400) [17:10:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:10:43] T313400: 2022-07-20 CloudVPS unstability after network outage - https://phabricator.wikimedia.org/T313400 [17:16:30] There has been a report in #wikimedia-tech of logins to toolsadmin.wikimedia.org not working. I am starting to investigate that now. [17:43:19] !log admin `sudo service striker restart` on labweb1001 [17:43:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:45:00] !log admin `sudo service striker restart` on labweb1002 [17:45:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:48:46] Restarting the services seems to have fixed things, but I need to spend more time trying to figure out why they broke. The application logs had no clues. [18:02:40] !log admin things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 (T313400) [18:02:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:02:43] T313400: 2022-07-20 CloudVPS unstability after network outage - https://phabricator.wikimedia.org/T313400 [19:31:00] !log tools reboot toolserver-proxy-01 to free up disk space probably held by stale file handles [19:31:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL