[09:45:26] Lunch [12:34:52] gehel: are you around [12:38:44] In the train, available on IRC only [12:39:13] ejoseph: what can I do for you? [12:43:40] Oops, I just realized we had our 1:1. So sorry [12:46:39] yh [13:10:23] greetings [13:36:03] o/ [13:47:44] dropping off kids, back in ~20 [14:02:07] back [14:12:06] o/ [14:24:59] \o [15:00:10] o/ [15:13:29] sigh... the wikidata statements we use to de-boost some items in search nevery really worked... we send a term query on an lowercase_keyword analyzed field with terms like P31=Q123 [15:22:34] :S [15:23:09] need better testing environments? But those kind of tests in cirrus are always super flakey. Almost need to be able to create an index-per-test (but then slow-slow-slow) [15:23:48] yes... [15:24:15] I think we should have detected that when we first deployed this I suppose [15:24:43] makes me feel that we manually tuned this with some bold assumptions that we never actually measured/tested [15:24:44] yea, we might need to start thinking about having some post-deploy test procedures to verify things work. I think we (certainly I) tend to trust things that appear to work [15:25:15] i guess that falls under "definition of done" that we don't really have [15:25:26] true [15:26:14] I wonder if we should extend what you've done for the mjolnir deploy, testing few obvious and well-known queries [15:26:16] i suppose i was thinking about that a little after working a little with the frontend team to add searchToken to autocomplete referrers, they have a separate QA person that the give testing instructions to that verifies things work [15:26:56] indeed that would force us to write down clear AC [15:27:52] extending the mjolnir tests might work [15:31:04] will write a ticket about that [15:39:16] I was supposed to follow up on a ticket with releng but it doesn't seem to be assigned to me, anyone remember what that ticket number was? [15:41:53] inflatador: was resolved I guess? T311370 [15:41:53] T311370: Positive test reports by Cindy-the-browser-test-bot should not result in setting Attention to the patch owner - https://phabricator.wikimedia.org/T311370 [15:55:54] Ah! Thanks dcausse [16:11:37] workout, back in ~40 [16:48:23] back [16:50:22] dinner [17:08:08] ryankemper or anyone else, looks like we're putting the s3_username into codfw elastic hiera only? Do we need to put it other places too? Re: https://gerrit.wikimedia.org/r/c/operations/puppet/+/807623/14/hieradata/role/codfw/elasticsearch/cirrus.yaml#165 [17:08:26] or is that just so we can start with codfw and do eqiad later? [17:15:11] inflatador: just first instinct, not actually looking closely, it does seem like that should go for all datacenters and not just codfw [17:26:12] ebernhardson Agreed, guessing it was put there to roll out the changes slowly. We'll meet in about an hour to do more of this stuff, I guess we can figure it out then [17:28:43] I thought we were just gonna xfer from codfw -> cloudelastic and be done [17:28:50] In which case we don’t need eqiad [17:29:01] yes, but :) [17:29:19] I guess the idea is we had to do this once before, but then undeployed it because it wasn't sustainable. But then we needed it a second time in less than a year [17:29:22] But that depends if we were gonna leave the creds in there or if it’s just a one-off [17:29:37] so my hope was to deploy a more permenant solution that we can use in the future without having to figure out how, just following the existing documented procedure [17:29:50] Yeah that makes sense ebernhardson [17:34:50] yeah, it should be rolled out to all hosts, but we can just do one DC to start. Still need to add the secret key into private puppet as well, I think '/srv/private/hieradata/role/common/search' is the right place, but LMK if not [17:37:36] lunch, back in time for pairing [18:19:44] back [18:43:37] gehel will you make it to pairing today? [19:20:00] oh right, Pink Floyd. Shine on, you crazy diamond!