[14:04:34] it will just be T377853 vs T378584 [14:04:35] T377853: RAID monitoring on new hardware spec requires new or updated user space cli tool - https://phabricator.wikimedia.org/T377853 [14:04:35] T378584: Evaluate hw-raid controllers for Supermicro's Config J - https://phabricator.wikimedia.org/T378584 [14:05:01] and probably just me understainding what has happened while I was out, see you later [15:03:43] jynus: so, in T384003 we tested the controller that the SM Config-Js came with; in JBOD mode it proved (eventually) possible to hot-swap a drive providing you then do: start initialization ; stop initialization ; set jbod. In the mean time, we eventually got a new controller from SM which we put into ms-be1091 and found that JBOD disk swapping Just Works (see T391854 ); so we concluded given the negligible cost differential we'd use [15:03:44] newer controller for SM Config-Js in future, and would also retro-fit the newer controller to the existing ms-be and thanos-be nodes that are SM Config-J. Puppet should arrange to deploy storcli appropriately now (T393146). [15:03:44] T384003: Perform fake disk swap on ms-be2088 as test - https://phabricator.wikimedia.org/T384003 [15:03:44] T391854: Swap RAID controller on ms-be1091.eqiad.wmnet - https://phabricator.wikimedia.org/T391854 [15:03:45] T393146: Add support for Broadcom RAID controllers using storcli - https://phabricator.wikimedia.org/T393146 [15:04:03] This is all just for JBOD, we didn't look at RAID at all. [15:04:23] I see, so for me, that was a "new new" controller [15:04:52] as I wasn't aware of the lastest part, which I assume happened while I was out [15:05:11] all good to me [15:05:41] 👍 [15:05:54] thanks for the explanation [15:06:15] NP; there are a couple of backup nodes that are SM Config-J, they are not going to be retrofitted (I mean, unless you want them to be :) ) [15:07:51] yeah, that's ok, as they worked ok for me, but I won't use them anyway for critical operation [15:08:21] do you know if the latest controller also uses storecli, or how that works? [15:08:44] or that will be handled by infra anyway and I shouldn't worry [15:13:56] AIUI, yes, it works with storcli and puppet will install that [15:16:59] thanks [15:39:15] Emperor: you didn't do any magic at T393049, right? [15:39:16] T393049: Multiple files returns "File not found: /v1/AUTH_mw/wikipedia-commons-local-public" error instead of showing correct file - https://phabricator.wikimedia.org/T393049 [15:44:58] don't think so, two ticks [15:45:48] jynus: no, I've made no changes to swift on that ticket, only inspected the state of what is already present [15:46:22] thanks, so I think I was the one to fix it [15:46:27] but I am not 100% sure [15:46:41] as the upload script fails, but works [15:46:52] :lolsob: [15:47:15] yeah, I would consider that a bug, but I am not going to make questions if it closes the ticket [15:47:47] I am just confirming the file is there on both dcs [15:48:55] actually, would it be easy to confirm it appears on upload logs, for example, for: swift list --prefix='2/21/Yankees_Baseball_(1)_(10562830654).jpg' wikipedia-commons-local-public.21 [15:49:15] of if you tell me a command, I can try to search for it [15:49:35] and the next time I can document the procedure [15:51:54] yeah, it is back on both dcs [16:00:32] swift list is what I'd use to check for presence, yes [16:01:16] yeah, I did that [16:02:09] I resolved it, http posts will only be needed in case you wanted to prove that they were actually uploaded :-D [16:02:39] because mw log was very confusing [16:03:39] I am going to be happy because at least recovery worked, so I will ignore all other bugs :-D [16:05:06] heh [16:05:44] [I could produce a rune to grep the swift proxy logs for the uploads, but I don't think it's worth it given the upload has demonstrably worked] [16:05:48] staring for too long at the black void of mw file handling can result in blindness [16:05:58] Emperor: I agree [16:06:15] It was mostly to convice me I wasn't crazy and the files were indeed missing [16:06:29] if you also saw that, I am ok :-D