[09:52:10] dcausse: I am trying to investigate, why a weighted_tag change for testwiki didn’t make it into ES. I can see the SUP intermediate event in kafka and it’s targeted at the omega cluster. But if I query one of the cluster omega nodes, the testwiki_content index doesn’t even exist. How would you query the CODF ES cluster? [09:53:29] pfischer: ouch [09:54:15] Special:Search returns results, so it must exist somewhere. [09:54:40] I guess I’am not querying the right host [09:54:49] you can query the clusters with https://search.svc.[eqiad|codfw].wmnet:9[246]43 [09:55:14] the port is what distinguish between psi, omega and chi [09:56:11] can't remember what is what, lemme find the config [09:57:45] according to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/ProductionServices.php#155 [09:58:13] 9243 is chi, 9443 is omega and 9643 is psi [09:59:31] Hm, neither of them shows testwiki_content under /_cat/indices/* [09:59:43] if the update pipeline thinks testwiki is omega then curl curl https://search.svc.codfw.wmnet:9443/testwiki_content/_alias?pretty should print something [10:00:18] pfischer: I see it: curl -s https://search.svc.codfw.wmnet:9443/_cat/indices | grep testwiki_content [10:00:25] testing eqiad [10:01:02] same I can see testwiki_content_1728059619@eqiad-omega and testwiki_content_1727943621@codfw-omega [10:03:37] ah, my bad mixed ports while forwarding… [10:04:42] dcausse: 1:1? [10:04:54] gehel: oops joining [10:04:57] https://meet.google.com/meg-wrep-dru [10:07:24] dcausse: thanks! [10:40:00] dcausse: So I was able to PUT a weighted tag into testwiki page 160445 and to retrieve it with the query ?cirrusDumpQuery shows. However, Special:Search still does not return a result: https://test.wikipedia.org/w/index.php?search=pageid%3A160445+hasrecommendation%3Alink&title=Special:Search&profile=advanced&fulltext=1&ns0=1 - How can I see which cluster the search is targeting? After all, search returns a page by just [10:40:00] querying by pageid. [10:43:18] pfischer: not sure I understand, how did you check that the tag was in the doc? cirrusDumpQuery should dump the query not the doc nor the results or of the query [10:43:49] I don't see tags in: https://test.wikipedia.org/wiki/Apollo_11?action=cirrusDump [10:44:02] this might be hitting eqiad [10:44:41] I see it in codfw [10:45:39] you can use the wikimedia debug extension to switch between DC switching between k8s-mwdebug-eqiad and k8s-mwdebug-codfq [10:45:48] s/q$/w [10:46:51] the link you pasted works when selecting k8s-mwdebug-codfw [10:56:41] dcausse: I tested Special:Search with `pageid:160445 hasrecommendation:link` since that didn’t return anything, I attached ?cirrusDumpQuery to get the query and requested that directly from the ES cluster. Since that didn’t return anything either, I manually added the weighted tag via PUT request to that same ES cluster. [10:56:52] But why would it only work against a debug host? [10:58:32] pfischer: mediawiki generally runs in multi-DC mode, which means you target the closest DC from your location, for us in europe it's generally eqiad [10:59:00] when hitting mediawiki@eqiad CirrusSearch will generally hit elastic@eqiad [11:00:09] with the wikimedia debug extension you can select what hosts you will target and thus is a way to forcibly run in codfw even if it's not the closest DC to you [11:00:52] the fact that it's a debug host does not matter much it's more the fact that the debug host you're hitting is from the DC you want to run your test query [11:01:44] here you manually put the tag in the test_content index running in codfw apparently but not in the index served by the cluster in eqiad [11:05:44] errand+lunch [11:08:50] Alright, TIL… thank you dcausse! [15:04:17] \o [15:10:51] o/ [18:25:32] * ebernhardson wonders where icinga::monitor::elasticsearch::cirrus_settings_check is supposed to go in the prometheus world [20:20:55] turns out there is still a benefit to invoking pcc from the python script instead of from gerrit, the `check experimental` on gerrit has to go through the queue, pcc.py skips the queue and runs it directly [21:16:32] the size of this diff is ... a bit scary :S https://puppet-compiler.wmflabs.org/output/1090529/4507/relforge1003.eqiad.wmnet/index.html [21:36:56] ouch [22:18:03] makes me a little less certain in this approach :P [22:18:30] like...most of that isn't ensure => absent, it's going to leave stuff all over....will have to ponder