[10:11:32] <Guest21>	 hi, i am having problems with my nullpool implementation with flask sql alchemy. Every now and then the connection to the database braks. Is there an example implementation or a toolforge project that has a nullpool implementation that i could look at?
[10:13:24] <Lucas_WMDE>	 Guest21: I’m not sure what a nullpool is, but I’ll just mention that persistent connections are not permitted, since it sounds like you might be trying to do something like that https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connection_handling_policy
[10:14:04] <Lucas_WMDE>	 (hm, though https://docs.sqlalchemy.org/en/14/core/pooling.html suggests nullpool is how you *disable* pooling, so maybe you’re trying to do the right thing?)
[13:54:36] <taavi>	 !log tools enable distro-wikimedia unattended upgrades T290494
[13:54:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:54:38] <stashbot>	 T290494: Revisit Toolforge automated package updates and version pinnings - https://phabricator.wikimedia.org/T290494
[13:56:48] <Rook>	 !log paws upgrade pywikibot on container start 437f46a1caf56cf860f85211af4567735410a621
[13:56:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[13:59:11] <taavi>	 !log admin manually attached [[User:Dreamy Jazz]] to wikitech for a password reset (https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Manually_associate_an_LDAP_account_with_wikitech)
[13:59:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:18:29] <taavi>	 !log tools depool tools-k8s-worker-42 for experiments
[15:18:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:26:04] <taavi>	 ^ uncordoned
[15:41:16] <andrewbogott>	 !log rebooting cloud*-dev for T307668
[15:41:17] <stashbot>	 andrewbogott: Unknown project "rebooting"
[15:41:26] <andrewbogott>	 !log admin rebooting cloud*-dev for T307668
[15:41:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:42:31] <dcaro>	 ^ does that include cloudbackup1*-dev stuff?
[15:42:53] <dcaro>	 (and not cloudbackup2*<with no -dev>?)
[15:55:25] <andrewbogott>	 dcaro: yes
[16:06:00] <dcaro>	 okok, just making sure (I have made that mistake at some point)
[16:13:39] <andrewbogott>	 dcaro: I'm trying to get the rabbit cluster to stand back up on cloudcontrol200x-dev. I'm unclear what's happening so far, 2004 says it can't reach 2001
[16:14:12] <dcaro>	 andrewbogott: looking
[16:14:23] <andrewbogott>	 docs for this are approximately https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Rabbitmq
[16:15:33] <andrewbogott>	 doesn't seem to be firewall although I always suspect ipv4 vs ipv6 firewall things
[16:19:38] <dcaro>	 I see this in the logs: 2022-05-10 16:17:58.306 [warning] <0.274.0> Error while waiting for Mnesia tables: {timeout_waiting_for_tables,['rabbit@cloudcontrol2003-dev'],[rabbit_queue,rabbit_durable_queue]}
[16:19:54] <dcaro>	 seems like it fails before starting up, trying to load the DB from disk, maybe it got corrupted?
[16:21:55] <dcaro>	 hmm... no, that seems related to the clustering
[16:22:15] <andrewbogott>	 yeah, I already went through a cycle of trying to delete the mnesia tables and start over. Possibly I made it worse, but that's the same issue I saw before I stopped everything and wiped out the tables
[16:23:12] <andrewbogott>	 It's probably worth doing apt --reinstall and letting it regenerate all of the support files and such in case there's still corruption someplace
[16:23:19] <andrewbogott>	 but I won't do that until you're standing clear :)
[16:23:28] <dcaro>	 okok, I'm fooling around a bit
[16:24:25] <dcaro>	 interesting: AMQPLAIN login refused: user 'nova' - invalid credentials
[16:26:32] <andrewbogott>	 oh... maybe my attempt to clean up wiped out the nova account?  But that shouldn't prevent clustering
[16:37:53] <andrewbogott>	 dcaro: ready for me to wipe things out or are you still digging?
[16:37:56] <dcaro>	   * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
[16:37:58] <dcaro>	   * can't establish TCP connection to the target node, reason: timeout (timed out)
[16:38:30] <dcaro>	 I mean, I'm ready xd
[16:38:43] <andrewbogott>	 hmmmm maybe it is firewall?
[16:39:19] <dcaro>	 25672 is not in the firewall
[16:39:21] <andrewbogott>	 that port isn't mentioned in ferm at all
[16:41:15] <dcaro>	 5672 is there though
[16:41:38] <dcaro>	 yep, it was a mistake in a diff reshuffling things
[16:41:44] <dcaro>	 it was there too
[16:43:08] <dcaro>	 andrewbogott: for your review https://gerrit.wikimedia.org/r/c/operations/puppet/+/790725
[16:43:23] <dcaro>	 (/me looking if there's any other missing ports from there xd)
[16:43:46] <dcaro>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/787003 xd
[16:43:50] <dcaro>	 it had a comment ;)
[16:44:00] <dcaro>	 but in draft... xd
[16:44:47] <andrewbogott>	 oh dang
[16:45:00] <andrewbogott>	 well, that explains a lot :)
[16:45:05] <dcaro>	 I think that until you reboot ferm does not remove rules
[16:45:17] <dcaro>	 (similar thing happened with cloudnets)
[16:45:24] <andrewbogott>	 yeah, that totally fits everything
[16:45:36] <andrewbogott>	 go ahead and merge? And then I'll try to repair whatever damage I did while troubleshooting
[16:45:52] <dcaro>	 👍
[16:46:21] <dcaro>	 you want me to run puppet on the hosts? or should I wait until you get rabbit sorted out?
[16:48:41] <andrewbogott>	 I'm doing it
[16:48:47] <dcaro>	 ack
[16:49:26] <dcaro>	 oh, you rebooted something?
[16:49:49] <dcaro>	 (lost connection to cloudcontrol2004-dev)
[16:50:13] <andrewbogott>	 oh yeah, sorry -- just trying to get things back into an expected state (it was complaining about mnesia not running)
[16:51:25] <dcaro>	 no prob
[16:51:42] <dcaro>	 fyi. 2003 still does not have the fw rules (2001 does)
[16:52:00] <taavi>	 oops. sorry about that!
[16:52:20] <dcaro>	 what happened?
[16:52:43] <taavi>	 missing that port when shuffling things
[16:53:01] <taavi>	 (just now reading the backlog and noticed that it was my patch that broke things)
[16:54:09] <dcaro>	 aaahhh, no problem :), thanks a lot for the patch, stuff happens, keep them coming ;) 
[16:54:38] <andrewbogott>	 2004 is working now, 2003 is having some sort of fit, I might just reboot it too if that won't mess with you
[16:54:54] <dcaro>	 andrewbogott: it does not have the firewall rules applied, did puppet run?
[16:55:03] <andrewbogott>	 puppet won't run
[16:55:06] <andrewbogott>	 I mean, it h angs
[16:55:11] <dcaro>	 oh
[16:55:13] <dcaro>	 that's not nice
[16:55:23] <dcaro>	 I don't see a puppet process
[16:55:34] <andrewbogott>	 and now I'm terminal there is getting all jittery
[16:55:45] <andrewbogott>	 ok if I reboot?
[16:55:46] <dcaro>	 works ok for me
[16:55:51] <dcaro>	 ok for reboot too
[16:56:07] * andrewbogott type 'reboot' sees 'ret' in the terminal
[16:56:25] <dcaro>	 hmmm... maybe you have connectivity issues?
[16:56:33] <dcaro>	 (or the path your ssh connection takes)
[16:58:17] <dcaro>	 I was able to run puppet, and it applied the ferm rule
[16:58:32] <dcaro>	 (took a long time trying to start rabbit first though, maybe some ordering would help)
[16:58:35] <andrewbogott>	 dunno, all my other terminals (on other cloudcontrols) were still responsive
[16:59:01] <dcaro>	 hmm...
[16:59:01] <andrewbogott>	 oh, heh, I thought it rebooted but my terminal messed up the mollyguard thing
[16:59:13] <andrewbogott>	 it's behaving ok for the moment
[16:59:44] <andrewbogott>	 Yep, I see it.
[16:59:49] <dcaro>	 okok
[16:59:54] <andrewbogott>	 This is where I'm getting stuck:
[16:59:56] <dcaro>	 the machine seems ok to me
[16:59:57] <andrewbogott>	 https://www.irccloud.com/pastebin/QozJdT5o/
[17:00:03] <andrewbogott>	 but I'll just be patient
[17:01:03] <dcaro>	 aaaahhh, yep, that will probably not work, you can try stopping the service and starting it again though, itś strying to start, on the `waiting for Mnesia` bit
[17:01:12] <dcaro>	 4 more tries until it gives up
[17:01:21] <dcaro>	 (2min)
[17:01:31] <andrewbogott>	 ok
[17:01:51] <andrewbogott>	 and now my terminal is messed up again
[17:01:59] <dcaro>	 that is weird :/
[17:02:12] <dcaro>	 2001 and 2004 see each other though, that's good
[17:02:48] <andrewbogott>	 ok, there, I finally got it to reboot
[17:02:53] <andrewbogott>	 perhaps it will make sense after that
[17:03:32] <dcaro>	 :crossingfingers:
[17:07:37] <dcaro>	 it did not join the cluster yet
[17:08:00] <dcaro>	 same issue
[17:08:05] <dcaro>	 (it seems)
[17:10:05] <andrewbogott>	 yeah, I'm trying to reinstall the package but it wont' reinstall until it can stop and it's trying...
[17:12:29] <dcaro>	 okok, probably zapping it (reset) and forcing it to re-join might be enough
[17:14:35] <andrewbogott>	 I can't get it to do anything but hang but I'll let you give it a try :)
[17:15:22] <dcaro>	 okok, on it
[17:20:08] <dcaro>	 oooohhh, I got the same issue with the console, when I Ctr+C while stopping the app
[17:20:24] <andrewbogott>	 that fits with what I saw
[17:20:30] <andrewbogott>	 interesting
[17:20:56] <dcaro>	 reset did not help
[17:22:15] <andrewbogott>	 maybe kill everything, rm -f all files, reinstall
[17:22:25] <andrewbogott>	 (although I don't know what 'all files' consists of specifically)
[17:25:04] <dcaro>	 ok, I moved the mnesia tables out, the server started, now we can reset it and rejoin 
[17:25:15] <andrewbogott>	 great, are you doing or shall I?
[17:25:21] <dcaro>	 on it :)
[17:25:24] <andrewbogott>	 cool
[17:26:50] <dcaro>	 okok, so the three are up
[17:27:02] <dcaro>	 still getting the credentials error xd
[17:27:38] <dcaro>	 I think puppet might be the one creating the users/auth?
[17:28:27] <dcaro>	 Notice: /Stage[main]/Openstack::Neutron::Rabbit/Rabbitmq::User[neutron-rabbituser]/Exec[rabbit_neutron_setup_perms]: Triggered 'refresh' from 1 event
[17:28:39] <dcaro>	 it's creating stuff
[17:29:48] <dcaro>	 okok, the errors stopped
[17:30:05] <andrewbogott>	 We might still need to create a trove user by hand for VM workers
[17:30:08] <andrewbogott>	 but otherwise I think we're good
[17:30:37] <dcaro>	 and I see no errors on the journal logs (from openstack stuff) either
[17:30:41] <dcaro>	 looks ok yep :)
[17:31:33] <dcaro>	 okok, gtg, let me know at the end of the day if you want me to continue anything tomorrow
[17:31:35] <andrewbogott>	 yep, creating
[17:31:45] <andrewbogott>	 ok! I need lunch and then will do some more reboots
[17:31:53] <andrewbogott>	 thank you for getting 2003 unstuck!
[17:31:56] <dcaro>	 good luck!
[17:32:03] <dcaro>	 np!
[17:48:58] <ori>	 Question re: <https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet> -- is it possible to proxy multiple backend ports? 
[17:49:18] <ori>	 or is it one port per hostname?
[17:53:53] <taavi>	 ori: one proxy hostname (something.wmcloud.org/wmflabs.org) is linked to one backend (instance+port), but you can have multiple proxy hostnames pointing to the same instance on different ports
[17:55:11] <ori>	 ok, that's what I figured. thanks
[17:58:12] <Jdlrobson>	 Possibly a stupid question - but  I've setup a cloud vps instance and I'm not sure which folder the default nginx  install is using to serve web pages
[17:58:50] <Jdlrobson>	 It was /var/www/html/ in debian-10.0-buster
[17:58:57] <Jdlrobson>	 but not sure where it is on  debian-11.0-bullseye 
[17:59:25] <bd808>	 Jdlrobson: maybe check the /etc/nginx/... config files?
[17:59:50] <Jdlrobson>	 I can't locate the nginx folder either
[18:00:17] <bd808>	 hmmm... but you did apply a puppet role or manually install nginx?
[18:00:48] <Jdlrobson>	 puppet role.
[18:01:54] <Jdlrobson>	 i haven't setup nginx manually but traffic seems to be going to nginx on https://pixel2.wmcloud.org/
[18:02:16] <bd808>	 that's probably the front-proxy nginx
[18:02:50] <Jdlrobson>	 ah okay, so it's not installed. I'm guessing  in debian-10.0-buster it was?
[18:02:58] <Jdlrobson>	 That would explain my confusion :)
[18:03:24] <Jdlrobson>	 (in the past I've simply set up vps instances and had a static server without any additional work)
[18:04:18] <bd808>	 Not sure how that would have happened for you honestly Jdlrobson unless there is some prefix puppet you have setup in the reading-web-staging project.
[18:04:34] <bd808>	 there is no default httpd in the base images we make available
[18:04:39] <Jdlrobson>	 Okay thanks for clearing that up
[18:04:59] <bd808>	 I don't see any puppet config at all on https://openstack-browser.toolforge.org/server/pixel.reading-web-staging.eqiad1.wikimedia.cloud which is the backend that pixel2.wmcloud.org points at.
[18:05:02] <Jdlrobson>	 When I setup skins.wmflabs.org/ it just worked out the box which may be very happy
[18:05:10] <Jdlrobson>	 I'm sure not sure that worked though
[18:05:53] <Jdlrobson>	 we're using es-dev-server for pixel but it doesn't survive reboots so I was curious what magic was powering skins.wmflabs.org
[18:05:56] <bd808>	 Jdlrobson: you have role::simplelamp2 applied on the backend host for skins -- https://openstack-browser.toolforge.org/server/skins.reading-web-staging.eqiad1.wikimedia.cloud
[18:06:18] <Jdlrobson>	 ahhh 
[18:06:51] <mutante>	 oh, someone uses it and was happy about it, yay
[18:07:03] <Jdlrobson>	 mutante: you made role::simplelamp2 ?
[18:07:09] <Jdlrobson>	 or added ?
[18:07:13] <mutante>	 yea, it was to replace simplelamp
[18:07:35] <Jdlrobson>	 well it's working so well i didnt even know i was using it :)
[18:07:43] <Jdlrobson>	 okay that was the missing piece of the puzzle i needed!
[18:07:44] <mutante>	 :)
[18:07:47] <Jdlrobson>	 thank you bd808 
[18:07:51] <Jdlrobson>	 and mutante for putting it there :)
[18:08:07] <bd808>	 yw
[18:08:31] <mutante>	 you can use prefix/project puppet if you want it on every instance in a certain project
[18:09:07] <Jdlrobson>	 When making puppet changes I have to soft reboot?
[18:10:23] <mutante>	 no, just run the puppet agent or wait
[18:10:48] <mutante>	 sudo puppet agent -tv
[18:11:35] <taavi>	 `sudo run-puppet-agent`*
[18:11:37] <Jdlrobson>	 Might be worth adding a page for static hosting on  https://wikitech.wikimedia.org/w/index.php?search=static+hosting&title=Special%3ASearch&go=Go&ns0=1&ns12=1&ns116=1&ns498=1 ?
[18:15:36] <mutante>	 technically it's not just static because this also installs PHP but I can't think of a better name
[18:16:05] <bd808>	 We are in general light on Cloud VPS specific Help: documentation. I agree that a little page on setting up a LAMP stack on an instance would be helpful.
[18:16:32] <bd808>	 oh... https://wikitech.wikimedia.org/wiki/Help:LAMP_instances
[18:16:38] <bd808>	 maybe needs some SEO help :)
[18:17:28] <bd808>	 found that with https://wikitech.wikimedia.org/w/index.php?search=httpd&title=Special:Search&profile=advanced&fulltext=1&ns12=1
[18:18:40] <mutante>	 simplelamp2 (httpd, memcached, PHP, mariadb), simplelap (same but without mysql/maria) and simplestatic (really just httpd) are 3 different roles available
[18:18:48] <mutante>	 that page is nice
[18:19:04] <mutante>	 we currently have a technical writer working on docs by the way
[18:19:23] <mutante>	 they were converting a "how to create VMs in cloud VPS" google doc
[18:36:28] <Jdlrobson>	 sweet
[18:52:00] <ori>	 does the proxy always map port 80 to the backend port? can i proxy a different port?
[18:52:16] <ori>	 right now i have public-hostname:80 proxying private-instance:6927
[18:52:25] <ori>	 i'd like to have public-hostname:6927 proxy private-instance:6927
[18:53:35] <taavi>	 not possible at the moment, only https on port 443 is proxied
[18:53:51] <ori>	 err, :443, right
[18:53:56] <ori>	 ok, I can live with that, thanks
[18:54:30] <ori>	 one more questionm re: https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet#Migrate_from_a_*.wmflabs.org_proxy_to_a_*.wmcloud.org_proxy
[18:54:43] <ori>	 are *.wmflabs.org proxy addresses deprecated / slated for eventual deletion?
[18:54:56] <ori>	 is there any advantage to using *.wmfcloud.org?
[18:55:01] <ori>	 *wmcloud rather
[18:55:11] <taavi>	 yes, but we don't have any concrete plans yet for removing the old ones
[18:56:16] <taavi>	 "labs" is a horrible term that we've been trying to get rid of since mid 2017 https://wikitech.wikimedia.org/wiki/Help:Labs_labs_labs
[18:57:08] <ori>	 yeah i know :)
[18:57:10] <ori>	 thanks
[18:57:24] * andrewbogott waves to ori
[18:57:39] <ori>	 hello!
[19:05:38] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:05:38] <wm-bot>	 !log admin Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:05:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:05:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:06:25] <wm-bot>	 !log admin Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
[19:06:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:29:44] <wm-bot>	 !log admin Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:29:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:33:06] <wm-bot>	 !log admin Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
[19:33:06] <wm-bot>	 !log admin Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
[19:33:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:33:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:33:22] <taavi>	 didn't we move all this spam to -feed?
[19:36:30] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:36:30] <wm-bot>	 !log admin Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:36:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:36:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:37:15] <wm-bot>	 !log admin Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
[19:37:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:39:26] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:39:26] <wm-bot>	 !log admin Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:39:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:39:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:40:41] <wm-bot>	 !log admin Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
[19:40:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:41:02] <bd808>	 taavi: the patch for that was merged, yes. Not sure that there were any tests to prove it worked though...
[19:44:02] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:44:02] <wm-bot>	 !log admin Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:44:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:44:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:45:02] <bd808>	 ori: $NAME.wmcloud.org proxies will also resolve as $NAME.wmflabs.org for backwards compat when migrating. We mean to disable creating new *.wmflabs.org names soon. (We actually just forgot that we had not previously.)
[19:45:54] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:45:54] <wm-bot>	 !log admin Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:45:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:45:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:46:34] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:46:34] <wm-bot>	 !log admin Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:46:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:46:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:47:24] <wm-bot>	 !log admin Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
[19:47:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:55:54] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:55:54] <wm-bot>	 !log admin Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:55:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:55:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:57:27] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:57:27] <wm-bot>	 !log admin Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[19:57:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:57:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[20:00:31] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[20:00:31] <wm-bot>	 !log admin Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[20:00:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[20:00:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[20:01:57] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
[20:01:57] <wm-bot>	 !log admin Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
[20:01:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[20:02:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[20:05:09] <wm-bot>	 !log admin Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
[20:05:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[20:21:50] <mutante>	 taavi: did you just create new instances? in that admin-monitoring project?
[20:22:11] <mutante>	 ah, the user name is right there. nevermind me
[20:57:15] <balloons>	 !log admin-monitoring cleanup fullstack test instances after cloudcontrol reboots
[20:57:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL
[21:32:32] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[21:32:32] <wm-bot>	 !log admin Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[21:32:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[21:32:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[21:35:28] <wm-bot>	 !log admin Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
[21:35:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[21:40:05] <wm-bot>	 !log admin Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[21:40:05] <wm-bot>	 !log admin Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@buster
[21:40:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[21:40:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[21:43:02] <wm-bot>	 !log admin Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
[21:43:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL