[08:11:46] Emperor: re: T309715 and the new account, I don't have the time to followup this week, would you be able to ? [08:11:47] T309715: Create swift thanos account for Search platform team - https://phabricator.wikimedia.org/T309715 [08:14:41] Yeah, I can try and get to that. Not that my week is un-busy, you understand :) [08:15:14] hehe I do, thank you ! [09:32:39] Emperor: good morning. What do you want to do with the sre.swift.convert-ssds gerrit patch? Wanna try it like this or do you need any change/adjustment? [09:36:51] volans: it looks good to me; shall I test it on the next few swift hosts I was about to reimage anyway? [09:37:03] [which is probably easiest if you've merged that CR?] [09:37:19] if you want sure! I'm not getting your last comment [09:37:41] do you wan tto add some commands to run to chekc that the host is ok to be taken down? [09:38:00] volans: I think I'm happy with "operator should confirm" [09:38:52] ok great, let me merge and deploy, if you want me to run the first one (there might be some silly mistake) let me know, happy to do it if that helps [09:39:30] volans: how about I grab you on IRC before my next reimage and I'll see if 🔥 results? [09:39:46] wfm :) [10:30:00] ...of course now I've hit the HP boxes [10:30:13] lol [10:38:55] don't worry, there are plenty more Dells to go 😿 [12:34:39] Le sigh, this HP system is really not wanting to come back up with disks and filesystems in order :( [12:43:29] es2031 just rebooted, possibly crashed [12:44:18] /dev/sdb3: LABEL="swift-sda3" _and_ /dev/sda3: LABEL="swift-sda3" [12:44:25] * Emperor not going to space today :( [12:53:39] I've opened T309977 for db2031, *I'm not attempting to restart mysql* [12:53:39] T309977: es2031 crashed (es2) - https://phabricator.wikimedia.org/T309977 [13:03:04] hello all! Any chance someone could look at https://phabricator.wikimedia.org/T309715 today (swift thanos acct request)? We need this access to restore a backup [13:11:36] it is on my radar, but I'm quite snowed under. Sorry :-/ [13:11:41] Will try and get to it [13:12:08] [how has account creation ended up on the critical path for a backup restore?] [13:13:31] Emperor np, thanks for looking. And yes, that's a legit question to which I have no answer ;( [13:15:43] FWiW, it's an env with no SLA/SLO, and is mostly working, but it is blocking our ability to upgrade [13:27:01] inflatador: you want an entirely new swift account? [13:28:16] inflatador: do you want to suggest a name for it? you can see hieradata/common/profile/thanos/swift.yaml for current names [13:28:36] [if you'd rather converse on-ticket happy to do so] [13:34:36] Emperor here is fine. Yes, let's make an entirely new swift account if that's OK. How about 'search_platform' for a swift account name? [13:41:33] works for me :) I'll get started (the process is a bit tedious - 3 different repos to update then a rolling restart) [13:43:51] Sorry to be a pest. If I can do anything else to help, LMK [13:47:34] https://gerrit.wikimedia.org/r/c/operations/puppet/+/803284 needs a +1 (maybe from godog if they're around) [13:50:41] Emperor: yep! +1'd [14:00:54] inflatador: should be done now. [14:01:27] Emperor excellent! Thanks again for your help [14:02:02] YW :) [17:04:25] Sorry to bug again: I'm having trouble logging in to the thanos-swift cluster with swiftclient. Can anyone confirm/deny that ST_KEY needs to be the password value as set in private puppet? More details at https://phabricator.wikimedia.org/P29441 [17:32:38] ah, looks like it might be a user issue. Reading thru https://wikitech.wikimedia.org/wiki/Swift/How_To [17:46:59] nm, got it! just needed to add a colon, exactly as it was in the PR: https://gerrit.wikimedia.org/r/c/operations/puppet/+/803284/2/hieradata/common/profile/thanos/swift.yaml [20:06:15] inflatador: if the docs could be improved, please do edit vigorously :) [20:06:19] [sorry, was AFK] [20:11:25] np, will see what I can do...I think it was more like I needed to RTFM a little closer ;)