[07:01:53] <Leaderboard>	 Hi, I have a project that is impacted by the Debian Bullseye migration. I am working on hopefully moving the project to Toolforge, but I will need some time (probably up to two to three weeks) as I just got my quota increase approved. I am wondering on how to request an extension or exemption? If it turns out that it isn't feasible to move to
[07:01:54] <Leaderboard>	 Toolforge, I intend to follow the OS migration. The "[REMINDER]: Migrate Cloud VPS Instances from Debian Bullseye" email I got points me here.
[07:19:44] <Leaderboard>	 !help (apologies if I shouldn't have done that)
[07:19:44] <wm-bot>	 If you don't get a response in 15-30 minutes, please email the cloud@ mailing list -- https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
[07:47:19] <dcaro>	 Leaderboard: can you open a phabricator task requesting the extension? We can then follow up there.
[07:50:40] <Leaderboard>	 OK thanks
[12:20:57] <isaacj>	 working for me now -- thanks dcaro !
[14:58:20] <wm-bb>	 <Yetkin> cannot shh to bastions...
[14:58:54] <dcaro>	 godog: ^ you doing reboots?
[14:59:50] <godog>	 dcaro: no
[14:59:53] <dcaro>	 ack, looking
[15:01:33] <godog>	 but yeah looks like bastion-15 is busted
[15:01:36] <godog>	 14:54 -wmcs-alerts:#wikimedia-cloud-feed- FIRING: InstanceDown: Project tools instance tools-bastion-15 is down   -            https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown 
[15:02:19] <dcaro>	 https://www.irccloud.com/pastebin/jyTSUMPQ/
[15:02:34] <dcaro>	 that's the last logs in the console, looking into dmesg (not sure what timestamp is that though xd)
[15:02:53] <dcaro>	 I think it's just out of memory
[15:03:00] <dcaro>	 https://www.irccloud.com/pastebin/UJGGCP12/
[15:03:21] <dcaro>	 let me check what's using it and I'll kill or reboot the VM
[15:03:47] <dcaro>	 hmm... OOMKiller should be doing it's thing though
[15:03:49] <godog>	 ack
[15:04:11] <dcaro>	 too unresponsive, let me reboot
[15:04:29] <dcaro>	 !log tools reboot tools-bastion-15, getting out of memory
[15:04:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:10:56] <godog>	 !log tools hard reboot tools-bastion-15 
[15:10:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:11:41] <godog>	 host is back
[15:14:48] <wm-bb>	 <Yetkin> Thanks man
[15:15:10] <godog>	 sure np!
[15:33:18] <Zppix>	 Question, is there an easier way to reimage an VPS instance besides deleting and recreating?
[15:37:47] <wm-bb>	 <harej> I think delete-and-recreate is the standard (VMs are cattle not pets)
[15:42:07] <Zppix>	 I figured, thanks
[15:43:10] <wm-bb>	 <harej> We have to do the same thing for InternetArchiveBot and we're at quota 😬
[15:43:21] <wm-bb>	 <harej> (You can request temporary quota bursts)
[15:43:49] <Zppix>	 that sounds fun... lol
[15:43:54] <Zppix>	 luckily I'm not even close to quota
[15:47:58] <mutante>	 create the new machine before deleting. rsync all the stuff between them. only then delete old instance?
[15:50:43] <Zppix>	 thanks
[15:52:37] <mutante>	 if you don't use puppet and want to transfer "which packages are installed" you can also do this:  dpkg --get-selections > somefile.txt | copy the file to new machine |  dpkg --set-selections < somefile.txt   and BOOM all the same packages are installed
[15:52:51] <mutante>	 of course the best fix is to use configuration management like puppet 
[15:58:08] <Zppix>	 yeah this time around I think I'll definitely setup a project-level puppet, thanks for the help mutante
[16:23:26] <mutante>	 yw, happy to answer questions about that if needed
[17:45:20] <bd808>	 Zppix: using volumes for storing data is really helpful too. a volume can be detached from your N-1 instance and reattached the the new N instance with a couple of clicks in Horizon. Much easier IMO than rsync or other copy mechanisms.
[18:00:54] <wm-bb>	 <harej> bd808: that very pattern has saved me so much time on my own cloud servers
[20:47:59] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: cab24068909455ccc3a11fb58066a1835d53c4024e9a32574b3894a2bcc3f767
[20:48:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL