[04:52:33] <Rook>	 andrewbogott: hi! The user data is kept under, I think, /srv/paws/project... Follow the data it's pretty large 
[04:53:01] <Rook>	 For the magnum ticket that seemed to get fixed when fcos was updated 
[04:53:37] <Rook>	 The patch that we were applying locally didn't seem to work so it can, probably, be removed. 
[08:42:37] <arturo>	 dhinus: thanks for your quick review yesterday on https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 I would like to get it merged this morning
[08:42:59] <arturo>	 but because the `move` blocks, will need separate merges for eqiad1/codfw1dev
[08:57:44] <dhinus>	 arturo: sounds good, I'll have another look in a moment
[09:34:37] <dhinus>	 arturo: can you run a "tofu plan" on codfw to check it's a no-op? the last tofu plan I see in the MR is old
[09:34:52] <dhinus>	 then I'm fine with the two separate merges
[09:49:07] <arturo>	 sorry, I was in a meeting
[09:49:16] <arturo>	 I'll create a MR only for codfw1dev
[09:50:14] <dhinus>	 ok!
[13:12:24] <arturo>	 FYI I gave chuckonwu 'owner' permissions on cloud/ repos on gitlab
[13:12:55] <arturo>	 let me know if you would rather have a more granular assignment 
[13:14:02] <dhinus>	 arturo: I think that's fine
[13:59:02] <arturo>	 thanks
[14:14:11] <arturo>	 quick +1 here ?  https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/167
[14:17:55] <andrewbogott>	 sorry about all the emails, trying to reduce the volume for the next round of reboots
[14:18:29] <dhinus>	 arturo: +1d
[14:18:39] <arturo>	 thanks!
[15:29:37] <andrewbogott>	 dhinus: have a minute to help me with a db schema issue? Trove seems to have a mistake in the db upgrade script and I'm having a hard time understanding the failure
[15:29:55] <andrewbogott>	 you can see it if you log into cloudcontrol1005.eqiad.wmnet and run 'sudo trove-manage db_sync'
[15:30:22] <andrewbogott>	 the migrations are in /usr/lib/python3/dist-packages/trove/db/sqlalchemy/migrations
[15:33:08] <arturo>	 :-(
[15:36:03] <andrewbogott>	 I have a work around prepared but somehow the failure is different in eqiad1 than it was in codfw1dev
[15:39:36] * arturo offline
[15:40:26] <dhinus>	 andrewbogott: looking
[15:41:46] <andrewbogott>	 AttributeError: Could not locate column in row for column 'values'  <- not helpful
[15:43:47] <dhinus>	 I wonder if it's a version mismatch problem
[15:44:11] <dhinus>	 I found https://stackoverflow.com/a/72921642/1765344
[15:45:00] <andrewbogott>	 possible. Same version in codfw1dev though and I got it to work there (although not without a hack)
[15:45:55] <andrewbogott>	 I would be fine with just applying the update by hand, it's not very many changes.
[15:45:58] <andrewbogott>	 haven't tried that yet though
[15:49:56] <dhinus>	 how would you apply it by hand? looking into the py migration file and manually translating to SQL commands?
[15:50:49] <dhinus>	 it might work but it's a bit tedious, you could then update whichever table is used to keep track of applied migrations
[15:51:44] <andrewbogott>	 yeah, it's not straightforward.
[15:51:47] <dhinus>	 I would first try playing with installing different sqlalchemy versions (with pip or some other way)
[15:51:52] <andrewbogott>	 Can you even tell from the error message which step is failing?
[15:51:55] <dhinus>	 and see if there's one where the script works
[15:52:29] <andrewbogott>	 ok. Looks like we used to be running 1.4, let's see if I can get that installed
[15:53:09] <dhinus>	 I'm not seeing which step is failing, no
[15:54:16] <andrewbogott>	 reverting to 1.4.46+ds1-1 seems to have worked!
[15:54:24] <andrewbogott>	 well, I think...
[15:54:33] <andrewbogott>	 does db_sync look good to you now?  It's so verbose...
[15:54:55] <andrewbogott>	 I wish a second run would just produce no output...
[15:55:27] * andrewbogott restarts the cookbook
[15:58:15] <dhinus>	 should I wait for you to run the cookbook, or can I run db_sync now?
[16:00:47] <andrewbogott>	 the host is rebooting now :)  so at least the cookbook thinks we're unstuck.
[16:00:55] <andrewbogott>	 Thanks! We'll see if trove still works after the upgrade, I guess
[16:01:48] <dhinus>	 nice! I will log off soon, but let me know if it's still broken and I can look later/tomorrow