[08:50:14] gehel: re T268648, I think it's still relevant since the current implementation is still querying the wikidata search API which we consider a bad design (MW calling itself) [08:50:15] T268648: [EPIC] MediaSearch should use a dedicated service/query for doing its concept-lookup instead of the wikidata search API - https://phabricator.wikimedia.org/T268648 [09:18:30] dcausse: ack [10:16:30] gehel: if you have a couple minutes: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/769075 [10:16:39] looking [10:16:50] thanks! [10:18:13] dcausse: looks good in principle, but there is much that I don't understand or lack context [10:18:25] is my +1 good enough for you to merge? [10:18:46] yes I just wanted a quick sanity check :) [10:41:11] early lunch (need to be back early for interview) [11:05:21] lunch [13:44:54] ryankemper I can confirm that it doesn't get written until the service restart [13:45:01] also, greetings [13:55:52] o/ [13:56:26] open hangout is now opened: https://meet.google.com/ugw-nsih-qyw [14:28:23] Following razzi suggestion, I've updated most of our meeting to use the same meet room starting next week. In case you want to bookmark it: meet.google.com/yau-mkip-tqg [15:55:25] heh, unexpected. Yesterdays test run of saneitizer finally ran 2 hours of jobs in 2 hours of wall-time. But that was during low hours, might as well still cut the indexing rate [15:55:59] (hugh has made more changes to k8s config relating to cpu throttling and such) [15:59:18] i don't trust these graphs though, we insert 200 jobs per second and process at a rate of 40 (no units). And somehow that keeps the queue empty. :P [15:59:37] Question: If I have Apple Dictionary searches made through search.wikimedia.org (cf. https://wikitech.wikimedia.org/wiki/Search.wikimedia.org) show up as webrequests with user_agent='AppleDictionaryService/131.2', shouldn't there be corresponding mediawiki_cirrussearch_request events in the same hour with http.request_headers['user-agent']='AppleDictionaryService/131.2'? [16:00:17] bearloga: hmm, depends how search.wikimedia.org works. I haven't looked at it since you were on our team, but i think it makes a request to api.php ? [16:00:53] if it does make a request to api.php then nothing in particular would get forwarded through since it's not a proper proxy. Seeing if i have any idea where that is [16:01:00] bearloga: err, actually i have a mtg now, but will check in 1hr [16:01:15] ebernhardson: thanks! I appreciate it [16:04:00] if the UA doesn't get passed to Cirrus Search that's fine, I was just looking for it as a way to double check that event.mediawiki_cirrussearch_request includes search.wikimedia.org searches [16:59:08] * ebernhardson wonders if it's intentional that the test case for apple's search endpoint looks for cthulhu [16:59:30] razzi: FYI we have a puppet deploy window in 30 mins (basically a dedicated window where brian/I pair with any devs on the team who need to get puppet/etc patches deployed) [16:59:46] razzi: rather than add you to the actual event i'll just drop this link here for later: meet.google.com/iqe-wcuz-mpn [17:00:13] i have a few, like most weeks :) https://gerrit.wikimedia.org/r/q/ownerin:search+project:operations/puppet+status:open [17:00:21] ryankemper: cool. Want to kick off the relforge upgrade now, or after that? [17:00:44] I guess the patch for the upgrade isn't quite ready yet, so no rush [17:00:57] hey all, there's a train blocker and we're being asked if it's on us? can someone take a look? https://phabricator.wikimedia.org/T303352 [17:01:26] ^ gehel [17:02:13] dcausse, ebernhardson: do you understand what T303352 is? [17:02:14] T303352: Wikimedia\Assert\ParameterAssertionException: Bad value for parameter $link: namespace must not be virtual - https://phabricator.wikimedia.org/T303352 [17:02:43] hmm seems to be the rest api? [17:03:20] there was changes to include redirect text recently maybe this is related? [17:03:30] razzi: yeah looks like we've got some comments of volans to address, so let's hold off till later, maybe during the SRE pairing session we have with gehel and inflatador in a few hours [17:03:49] Not sure if the "*" is the request url is the actual url failing or if this is a wildcard for all requests generating that error? [17:04:46] Which team is responsible for the REST API ? [17:05:23] platform engineering I think? [17:05:40] API Platform [17:05:43] I think so too. Not sure who to ping [17:05:50] I'll let them know, thanks [17:05:55] I'd tag API Platform and MediaWiki-REST-API [17:06:11] cool go for it [17:07:32] on the good news S3 seems to have helped making the flink app run again in k8s [17:07:39] \o/ [17:07:50] I'll ping Will on Slack about that error [17:08:52] bearloga: indeed the apple-search endpoint is a php script that curl's to our regular backend. You should find the user agent set to 'Wikimedia OpenSearch to Apple Dictionary bridge' [17:09:33] ebernhardson: thank you so much!!! [17:09:35] dcausse that is great new [17:09:40] s [17:09:45] quick workout, back in ~30 [17:10:17] dcausse: indeed, thats great news. things start working again :) [17:13:20] yes... quite a relief tbh :) [17:18:11] inflatador: razzi: Couple things: (1) It looks like the file that elasticsearch tries to create is `/etc/elasticsearch/elasticsearch.keystore.tmp`, so it's not sufficient for us to just set the perms on `/etc/elasticsearch/*` [17:19:23] inflatador: razzi: (2) Looks like we can just run a `sudo /usr/share/elasticsearch/bin/elasticsearch-keystore create`. The only thing I'm unsure of is, the keystore might still not be readable by the `elasticsearch` user upon startup, so that step might not obviate the need to set the permissions [17:20:28] Sounds like `sudo ... create` and then `chown` would work? [17:26:05] razzi: something like that, although I just realized there's a specific `keystore upgrade` command so that probably better replicates what elasticsearch is doing when we upgrade & restart the service [17:26:33] also I'm not totally sure if running the upgrade manually will make elasticsearch "realize" that it doesn't need to do the upgrade itself upon restart; the docs imply that it does but aren't explicit [17:27:03] guess we'll probably want to do some more testing on deployment-prep potentially [17:28:29] lost track of time and need to go eat some yogurt, will be a little late to puppet deploy window [17:41:28] back, sorry I'm late too. Will be there in ~3m [18:10:38] inflatador, ryankemper, razzi : I'll be 30' late for the SRE pairing session. Last minute reschedule of a call with parents of deaf children that I really don't want to miss [19:14:35] gehel : ACK . ryankemper and razzi are you OK with moving back 30m? Would also give me more time to make lunch [19:15:14] inflatador: razzi: yup [19:59:29] Finishing up calories, 5’ late [20:00:32] ACK [20:00:45] 5' late as well, sorry [21:34:15] * ebernhardson is for some reason not convincing local elasticsearch to log deprecation warnings [23:34:03] ahh, it's not really not giving them. But it's supressing duplicates in the logs and instead adding Warning headers to the responses [23:35:30] maybe we can find some way to generically notice those and log stack traces on our side, would make tracking down deprecations easier