[10:02:10] lunch [12:37:16] Hmm [12:37:42] Seems like geosearch is not giving any results anymore on the beta cluster o.O [12:40:04] inflatador: Maybe there's some underlying process broken and since you ran these scripts an Friday now the whole index is useless? 😅 [12:59:52] WMDE-Fisch Let's hope not! I'll take a look [13:00:00] :-D [13:01:18] At least this query was giving me results last Friday ... and not it's not: [13:01:30] https://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&format=json&formatversion=2&prop=coordinates%7Cpageprops%7Cpageimages%7Cdescription&colimit=max&generator=geosearch&ggsbbox=48.87397380289263%7C2.193660736083985%7C48.832181625698475%7C2.51089096069336&ggsnamespace=0&ggslimit=50&ppprop=displaytitle&piprop=thumbnail&pithumbsize=300&pilimit=50 [13:01:45] So, something happend ;-) [13:08:58] Are there any other queries giving you similar issues? I checked that job I ran Friday and there are no errors [13:20:57] So as I (roughly) understand it, geosearch gets its "content" via the GeoData extensions that implements the coordinates parser function https://www.mediawiki.org/wiki/Extension:GeoData#Parser_function [13:21:34] On the en wikipedia beta cluster there are some pages that use a template for that function. [13:22:15] So where ever you add a coordinate using that template/parser function, the search should index that so you can find it on a "nearby" search [13:22:38] e.g. https://en.wikipedia.beta.wmflabs.org/wiki/Paris [13:23:00] or [13:23:04] https://en.wikipedia.beta.wmflabs.org/wiki/Germany [13:24:01] I'm with you so far. Let's ask in #wikimedia-cloud [13:24:15] On Friday I could run API requests just looking for a search radius or a bounding box around each of these examples and I got results [13:24:30] Now I dont ^^' [13:25:22] inflatador: sounds good. I'll have to run now but might jump in later again [13:28:36] inflatador: how did you run the saneitizer? [13:29:02] dcausse invoked from deployment-mwmaint02 [13:29:15] without any args? [13:29:18] /usr/local/bin/mw-cli-wrapper /usr/local/bin/foreachwiki extensions/CirrusSearch/maintenance/SaneitizeJobs.php --push --refresh-freq=7200 [13:29:41] ok [13:30:09] this should not have hurt anything [13:30:19] I suspect a broaded issue with the beta cluster [13:30:28] s/broaded/broader [13:30:36] I was also creating new elastic nodes on Friday, but as far as I know none of them every joined the cluster [13:30:41] ...ever that is [13:31:03] yes I doubt they could see each each others [13:31:43] the enwiki index only has 16 docs which is highly suspicious [13:33:52] The docs get added via job queue? [13:34:07] something's not right, https://en.wikipedia.beta.wmflabs.org/wiki/BeforeEach-name-0.9959470321005754-I%C3%B1t%C3%ABrn%C3%A2ti%C3%B4n%C3%A0liz%C3%A6ti%C3%B8n is in the index but it's not found when querying https://en.wikipedia.beta.wmflabs.org/wiki/BeforeEach-name-0.9959470321005754-I%C3%B1t%C3%ABrn%C3%A2ti%C3%B4n%C3%A0liz%C3%A6ti%C3%B8n?action=cirrusDump [13:34:16] inflatador: yes [13:35:01] so the docs are there, but the querying mechanism appears to be broken? [13:35:48] plenty of docs are missing from the index and the few that are in cannot be retrieved [13:36:39] that's not good, is there a way to look at historical document count for an index? [13:37:23] I don't think we track this in prometheus for this cluster [13:38:00] oops scratch the number I gave [13:38:03] do you know if we dump the indexes? [13:38:22] docs are in elastic but can't be extracted [13:41:36] directly from elastic too, or just from the frontend? [13:41:54] from elastic I see them but not from mediawiki [13:42:31] I just merged the es68 branch so might be related to this [13:44:10] can't login to deployment-mwmaint02.deployment-prep.eqiad.wmflabs :/ [13:44:23] :0 is there a way to tell if it's affecting production in the same way? [13:45:12] inflatador: this branch has not landed to prod yet so trying to debug this out on the beta cluster to understand the root cause [13:45:38] dcausse cool, I started meet.google.com/dvx-apwd-wdk if you want to work together on deployment-mwmaint02 [14:13:07] why am i banned from wikitech [14:13:29] why [14:24:17] sigh... the transparent _doc -> index type conversion does only work for apis like get, _search fails... [14:24:58] I might revert the es68 branch merge into master while we figure out a fix [14:25:21] hi [14:26:18] guys [14:26:19] or fix the search apis to always target the index instead of the Type [14:27:38] or reindex all the wikis... [14:28:10] what is this for [14:28:38] why is this channel here [14:29:10] oi [14:30:30] guys [14:31:44] GUYS [14:32:43] hello [14:36:57] dcausse: :( [14:37:02] also, \o [14:37:04] o/ [14:37:20] yes not sure we can find a quick hack before the branch cut :/ [14:37:52] seems safest to punt for a week and figure this out, i guess it means cindy is missing a few tests too [14:38:05] cindy rebuilds the indices so it works [14:38:10] oh, right [14:38:19] yes let's revert and think about it [14:38:45] the transition is massively more painful without those aliases :( will take some testing [14:39:05] i guess having extra roundtrip's for non-search wouldn't be the end of the world, but messy [14:40:07] I was experimenting using Index instead of Type to avoid that during search requests [14:40:48] makes sense where possible, but when i looked there were a variety of methods on Type that weren't available on Index [14:41:06] yes :/ [15:58:45] errand, back later tonight [17:09:19] lunch/errands, back in ~1h [18:12:56] back [19:40:48] lunch [20:21:39] have to pick up my son today, back in ~45 [20:31:53] back [20:52:00] back [22:35:55] Good night!