[10:03:05] <dcausse>	 ran a quick bench to test the WAND optim with morelike, seems quite significant: https://phabricator.wikimedia.org/P96536
[10:04:39] <dcausse>	 this is using a 500 articles dataset running two variants: track_total_hits: true (the current default) vs track_total_hits: 4 (what would be served for RelatedArticles)
[10:05:15] <dcausse>	 p90 drops from 450ms to 250ms on enwiki
[10:05:50] <dcausse>	 lunch
[10:25:45] <btullis>	 I'm really sorry about this mix-up with these disks. I'm working right now on finding a better solution for you.
[13:23:40] <inflatador>	 <o/
[13:40:51] <dcausse>	 o/
[14:34:33] <ebernhardson>	 btullis: no worries!
[17:36:54] <dcausse>	 dinner
[19:50:37] <btullis>	 By way of an update, I am re-purposing three machines in eqiad for you. They were previously ml-cache100[1-3]. They each have 1.92 TB SSDs, 48 cores, and 256 GB of RAM. I believe that this will be sufficient for your experiment, although with 3 nodes the instance density is greater than with dse-k8s-worker100[1-4].
[19:51:11] <ebernhardson>	 btullis: i appreciate it! It cuts the available ram as well, but hopefully will work out.
[19:51:34] <ebernhardson>	 oh actually no, the other instances were 128G, so this is more ram
[19:54:57] <btullis>	 Ack. Yeah, fingers-crossed. So far, I've brought them back from having been decommissioned (they sat in their racks for 5 years, never having gone into service as Cassandra hosts) as ml-cache100[1-3].
[19:54:58] <btullis>	 Next I will rename them as dse-k8s-worker104[2-4] and add them to the dse-k8s-eqiad cluster, probably tomorrow morning.
[20:05:11] <inflatador>	 If I can help w/any of that LMK
[20:06:33] <btullis>	 Thanks. I just filed: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1346130 - inflatador if you have time to run this process https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimagingagainst those three hosts to rename them, that would be great.
[20:07:17] <btullis>	 ml-cache1001 and ml-cache1003 have just finished coming back out of retirement using their old names. ml-cache1002 should finish in a minute.
[20:07:38] <inflatador>	 👀
[20:09:07] <btullis>	 We shouldn't need the old manual method, just the `sre.hosts.rename` cookbook now, followed by the reimage cookbook.
[20:10:13] <inflatador>	 Unrelated, but here's some more takeaways from the DC switchover T438896
[20:10:14] <stashbot>	 T438896: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896
[20:10:14] <btullis>	 https://usercontent.irccloud-cdn.com/file/HV5YhLJU/image.png
[20:11:18] <btullis>	 I've got a subsequent patch that I will do tomorrow, that will take these two 1.3TB partitions and make an LVM volume group from them, which will be assigned to the `topolvm-balanced` storageclass.
[20:11:35] <btullis>	 For reals, this time.
[20:12:31] <btullis>	 inflatador: All three reimage cookbooks have completed. So if you get the time to rename and re-run, that would be great. if not, no worries.
[20:16:32] <inflatador>	 btullis OK, I'll merge the above and run the rename/reimage cookbooks
[20:18:56] <btullis>	 Cheers.
[20:19:53] <inflatador>	 assuming the internet stays up over here ;(