[08:00:48] <taavi>	 morning
[08:01:02] <taavi>	 looks like I have a bunch of cloudvirts to work with (thanks Andrew) so I'm just going to start moving stuff
[08:39:47] <arturo>	 ack
[11:07:14] <taavi>	 has anyone seen 'cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 1c5e694b-7771-4fa2-825e-9cb8c317f31a).' while resizing an instance before?
[11:08:33] <arturo>	 no :-(
[11:10:08] <taavi>	 no hits from phab either
[11:12:24] <dhinus>	 I don't remember seeing it before :/
[11:13:07] <taavi>	 and I see two attachments: https://phabricator.wikimedia.org/P65154
[11:13:24] <taavi>	 I guess I could remove the duplicate ('reserved'?) one, reset the state to shutoff, and then try the migration again
[11:13:28] <taavi>	 thoughts?
[11:14:48] <dhinus>	 makes sense, I would maybe try removing both and re-attaching after the migration, but not sure if it will help
[11:16:22] <dhinus>	 I don't remember if I ever resized/migrated an instance with cinder volumes attached, in theory it should work just fine but I wonder if there's some hidden bug
[11:25:55] <taavi>	 no, now it's back in ERROR
[11:45:29] <Amir1>	 Hi, Do I need to file a quota request even if I need some it for a couple of hours only to do the bullseye migration for codesearch? 
[11:46:47] <Amir1>	 https://phabricator.wikimedia.org/T367479#9902922
[11:47:05] <Amir1>	 need 80GB volume so I can attach it to the new vm
[11:47:44] <Amir1>	 OTOH, If I can attack a volume to two VMs, maybe I can try that too but I'm not seeing it in horizon
[11:47:50] <Amir1>	 *attach
[11:50:38] <taavi>	 seems like I got lost in a netsplit - did I miss anything?
[12:10:33] <arturo>	 taavi: this is what I have
[12:10:34] <arturo>	 https://www.irccloud.com/pastebin/Sfon78sG/
[12:10:59] <arturo>	 Amir1: yes, please, file a ticket
[12:19:17] <taavi>	 andrewbogott: when you're around, can I ask you to have a look at the stuck VM (fastcci-nfs-1.fastcci)?
[13:44:59] <andrewbogott>	 taavi: I'm looking but I'm also about to have a meeting so it might be a bit before I get anywhere
[13:45:31] <andrewbogott>	 taavi: also, if you don't need to go then after the meeting I'd like you to supervise me moving some VMs so i can help migrate stuff later in the day
[13:45:50] <taavi>	 sure
[13:49:15] <andrewbogott>	 taavi et al, what I did for that fastcci host was:
[13:49:23] <andrewbogott>	 openstack volume attachment list --all-projects | grep cf6dc802-f95c-48c1-a07b-8791185a5e45 (the instance id)
[13:49:32] <andrewbogott>	 and then
[13:49:40] <andrewbogott>	  openstack volume attachment delete <the attachment id I found>
[13:49:42] <andrewbogott>	 and then
[13:49:52] <andrewbogott>	 openstack server reboot --hard cf6dc802-f95c-48c1-a07b-8791185a5e45
[13:49:58] <andrewbogott>	 and now I'm putting it back together
[13:50:59] <andrewbogott>	 (and now that I've read the backscroll, I see that taavi already did 90% of the work there)
[14:52:45] <andrewbogott>	 Raymond_Ndibe: I just +1'd https://phabricator.wikimedia.org/T367878, they have some momentum with their OS migration so would be good to get it filled quickly
[14:52:55] <andrewbogott>	 (oh or maybe I'll just do it if he's out)
[14:53:15] <andrewbogott>	 taavi: how are you prioritizing projects for the OVS move?
[14:53:34] <taavi>	 andrewbogott: I'm going through the list in alphabetical order
[14:53:41] <andrewbogott>	 that's easy!
[14:53:48] <andrewbogott>	 btw can I get a second +1 for T367878?
[14:53:49] <stashbot>	 T367878: Extra 80GB volume to allow migration of buster VM to bullseye - https://phabricator.wikimedia.org/T367878
[14:54:34] <taavi>	 i just implemented it instead
[14:54:38] <andrewbogott>	 fair :)
[14:54:50] * andrewbogott backspaces over his command rather than hitting enter
[14:56:29] <andrewbogott>	 taavi, do I need to observe any selectivity about projects or does the cookbook already skip things with g2 flavors and other unmovable things?
[14:57:37] <taavi>	 it has an allowlist of flavors to migrate, so any weird flavors are totally fine
[14:57:49] <taavi>	 the only thing to be careful about at this point is magnum things
[14:58:51] <andrewbogott>	 ok. is there a checklist?
[14:59:30] <andrewbogott>	 Well, I should have started with this question: is this sufficiently automated that I should just leave it to you? I'm just trying to share the tedium :)
[14:59:37] <taavi>	 1. check that project does not have any magnum instances
[14:59:37] <taavi>	 2. run the wmcs.openstack.migrate_project_to_ovs cookbook
[14:59:37] <taavi>	 3. ???
[14:59:37] <taavi>	 4. profit
[15:01:24] <andrewbogott>	 ok, so e.g. sudo cookbook wmcs.openstack.migrate_project_to_ovs --cluster-name eqiad1 --task-id T326373 --project xtools 
[15:01:25] <stashbot>	 T326373: Migrate Cloud VPS to Neutron Open vSwitch agent - https://phabricator.wikimedia.org/T326373
[15:01:36] <andrewbogott>	 Oh, by 'checklist' I meant, 'how do I know how far along in the alphabet you are?'
[15:02:00] <taavi>	 please don't add a --task-id there, I don't want that to fill my mailbox
[15:02:24] <andrewbogott>	 ok :)
[15:03:14] <taavi>	 so far it's automated enough that I can deal with the standard cases by myself. so far I'm up to 'fr-tech-dev'
[15:04:03] <taavi>	 I think it'd be more helpful if you helped with the "things that worry me" list. so far that includes trove, magnum, toolforge nfs, and all the bastions
[15:04:56] <andrewbogott>	 ok, sounds good. I'll make a magnum test case
[15:05:09] <taavi>	 the other annoying thing that would help at this point is all the VMs with non-standard g3 flavors
[15:05:13] <andrewbogott>	 do you have specific reason to suspect magnum?
[15:05:15] * andrewbogott nods
[15:05:39] <taavi>	 magnum is configured to automatically create VMs, and I doubt that changing existing VMs will change what it'll try to create in the future
[15:06:08] <andrewbogott>	 true
[15:15:03] <andrewbogott>	 taavi: I think in most cases the right thing is to just redeploy after something like https://github.com/toolforge/paws/compare/main...ovs
[15:15:13] <andrewbogott>	 (assuming that I know how to do that)
[15:17:33] <andrewbogott>	 oh, and apparently I need Rook to give me r/w access to the quarry repo
[15:20:43] <taavi>	 andrewbogott: can you stamp https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/5?
[15:22:30] <andrewbogott>	 approved
[15:22:58] <Rook>	 andrewbogott: You're not an owner on the GitHub toolforge org?
[15:26:36] <andrewbogott>	 all I know so far is that I can't 'git clone git@github.com:toolforge/quarry.git'
[15:26:44] <andrewbogott>	 but I can w/paws
[15:26:46] * andrewbogott double checks
[15:27:26] <andrewbogott>	 oh wait, I can't do that either. So how did my paws checkout get here...
[15:27:44] <andrewbogott>	 Rook: sounds like you should ignore me for now
[15:29:50] <andrewbogott>	 OK, I'm sorted now
[15:42:43] <Rook>	 👍 
[15:45:17] <arturo>	 FYI experimenting with haproxy for k8s @ toolsbeta
[15:45:58] <andrewbogott>	 Rook: have a minute to help me deploy quarry? I copied quarry.key from your homedir to mine, ran 'git-crypt unlock ~/quarry.key'...
[15:46:09] <andrewbogott>	 deploying says
[15:46:12] <andrewbogott>	 https://www.irccloud.com/pastebin/vuZZ8dw5/
[15:53:34] <Rook>	 I will but probably not for 30 or 40 minutes 
[15:53:40] <arturo>	 taavi: I just experimented with this in toolsbeta, wdyt? https://gerrit.wikimedia.org/r/1047113
[15:53:58] <andrewbogott>	 Rook: at your convenience :)
[15:57:09] <taavi>	 arturo: that seem sfine at least at a glance
[15:59:20] <arturo>	 taavi: thanks, I will follow up tomorrow
[16:19:48] * arturo offline