[11:31:20] lunch [15:43:59] \o [16:04:54] dcausse, ryankemper, inflatador, Trey314159: Wednesday meeting has started: https://meet.google.com/yau-mkip-tqg [16:05:31] I'm in another meeting for about one hour still, will join at ~17 UTC [16:07:21] dcausse: ack [16:38:23] oops, phone wasn't properly plugged in overnight [16:38:25] will be right there [16:46:01] ryankemper: I dropped off to get a snack, but spicerack is deployed and we can give the cookbook a go whenever you're up for it [16:47:42] razzi: excellent [16:55:13] An idea for y'all on the search team: in data engineering we have a meeting url that we use for every meeting and for impromptu meetings that we call "the batcave", so it's always the same link and you can bookmark it. It saves us a bit of hassle [18:31:07] dinner [19:04:58] lunch [19:07:29] What i had mentioned about "google" issuing multiple search requests and cutting out old ones probably came from this, in it "google search" was more an example, but it's from rob pike who probably at least has a general idea of what they are doing: https://talks.golang.org/2012/concurrency.slide#50 [19:07:40] s/cutting out old/cutting out slow/ [19:08:19] the whole section from p42-50 covers it. There is certainly a video where he talks over these slides somewhere [19:19:57] * ebernhardson finds out Elastica has a request logger...wonder how we turn that on [19:49:38] ryankemper: I'm back from haircut / lunch, lmk when you're back! [20:03:23] interesting error message: Valid states are 'starting, running'. The machine is in the 'gurumeditation' state. Please verify everything is configured properly and try again. [20:22:36] razzi: cool almost done, I'll be ready in 10-15 mins [20:24:30] ebernhardson: hmm, probably should keep it in that `gurumeditation` state. We wouldn't want to harsh its vibe [20:26:51] mpham: I might be 3 minutes late to our meeting [20:28:41] apparently thats VirtualBox's way of saying 'its broken and you need a guru to fix it'. destroy and rebuild didn't help, but apt-get upgrade did the trick. No clue what it would have been [20:29:30] the upgrade did recompile some dkms modules, maybe something there [20:37:02] ebernhardson: just sanity checking, you're not already running kvm on the machine, right? (https://github.com/kubernetes/minikube/issues/4913) [20:37:12] razzi: ok, ready whenever [20:51:51] dcausse: (for tomorrow) is T268648 still valid? I think the implementation has changed enough that this is irrelevant now) [20:51:52] T268648: [EPIC] MediaSearch should use a dedicated service/query for doing its concept-lookup instead of the wikidata search API - https://phabricator.wikimedia.org/T268648 [20:55:52] ryankemper: not entirey sure, `lsmod | grep kvm` suggests yes :) [21:08:17] is it accurate to summarize second chance search as: when a search term results in zero results, we try to detect if it's another language and then search that wikipedia with the same search term [21:12:06] mpham: i think its < 3 results, but otherwise yes [21:12:38] mpham: the exact number isn't important i suppose, but we were trying to target poorly performing queries and not only zero results [21:22:39] lunch [22:06:15] back [22:11:33] ebernhardson: can you remind me why codfw elastic gets so little traffic (at least based on grafana)? [22:12:20] i know for wdqs for example it's a result of geoip, but not sure if that applies to elastic when all the requests are coming from mw appservers [22:14:01] ebernhardson: actually I may have just rubber ducked myself, are the mw appservers going thru geoip, meaning that they'll only be hitting either eqiad or codfw exclusively based on which dc is active for mw? [22:15:46] ryankemper: cirrus sends traffic to the cluster closest to the mediawiki application servers. mediawiki only runs in one dc, so only that dc has traffic [22:16:21] ryankemper: so codfw should typically have only update traffic, plus anything issued by mjolnir-msearch via analytics pipelines [22:16:45] ebernhardson: ack, thanks [22:25:48] * ebernhardson for some reason didn't remember that a bare run of phpunit against mediawiki is 36k tests [23:47:31] inflatador: (for tomorrow) wrt https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/769109/comment/7844bccd_17bd0a90/ my comment here, does the keystore file not get written till the actual elastic service restart? it sounds like from the docs that it's likely done during the actual apt command, which would obviate the need for us to wait till after the restart cmd to restore `root` as the owner