[08:45:28] <gehel>	 inflatador: can you move the phab task to in progress when you start working on it? ^
[08:55:41] <gmodena>	 o/
[08:55:51] <gmodena>	 checking the failing mjolnir dag
[09:01:07] <dcausse>	 o/
[09:31:55] <pfischer>	 o/
[09:33:17] <pfischer>	 dcausse: Regarding your comment https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1129871/comment/302780b2_b248fe10/ - Do you expect full tag replacement? Because right now it’s implemented to replace prefixes only.
[09:35:09] <dcausse>	 pfischer: what do you mean by "full tag replacement", the way you implemented the logic is what I was expecting so perhaps my review comment is misleading?
[09:36:35] <pfischer>	 Alright, the suggested settings.txt comment suggested it’s replacing tags but the tag names would remain unchanged. So I just make the documentation more explicit. Thanks!
[09:37:26] <dcausse>	 oh I see my example "old.tag.name", yes should be more like "old.tag.prefix"
[09:39:42] <dcausse>	 and "replacement tag" -> "new tag prefix name", yes definitely what I suggested is confusing, sorry about that!
[09:39:48] <pfischer>	 np, fixed, just wanted to make sure we expect the same thing here.
[09:41:29] <dcausse>	 I also often refer to "tag" as "classification.prediction.articlecountry" it's more accurate to say "tag group"/"tag prefix"...
[09:59:37] <pfischer>	 dcausse: Understood. But what’s the glossary terminology? <prefix>/<tag>|<score> ? (https://wikitech.wikimedia.org/wiki/Search/WeightedTags)
[10:03:23] <dcausse>	 pfischer: technically yes, but when speaking outside the team I'm not sure, saying "articlecountry prefix" might not be very explicit
[10:04:39] <pfischer>	 dcausse:  Ah, that makes sense, I forgot about the non-tech context. ^^
[10:04:39] <dcausse>	 or "articlecountry weighted tags" might be ok, the *s* in tags suggest some sort of grouping
[10:09:25] <dcausse>	 I think the problem might just be me not using a precise terminology in a technical context, outside of the team using "weighted tags" was generally not so ambiguous
[10:40:05] <gmodena>	 i did a bit of a mess with param passing in the data retention mjolnir changes :(
[10:40:13] <gmodena>	 patches incoming
[10:53:30] <dcausse>	 lunch
[12:08:28] <gmodena>	 workout+lunch
[13:13:18] <inflatador>	 <o/
[13:16:35] <dcausse>	 o/
[13:19:56] <inflatador>	 thanks for the review on that wdqs alerts patch...will get a new one up that uses your approach shortly
[13:21:05] <dcausse>	 thx!
[13:55:06] <inflatador>	 gehel our 1x1 is double-booked with the DPE deep dive, what do you want to do?
[13:57:02] <gehel>	 inflatador: let's cancel our 1:1 for today
[13:57:05] <inflatador>	 gehel ACK
[14:12:05] <ebernhardson>	 \o
[14:12:36] <inflatador>	 .o/
[14:13:48] <dcausse>	 o/
[14:55:52] <inflatador>	 gehel are we doing the DPE retro in 5m, or triage in an hr, or both?
[15:51:52] <ebernhardson>	 hmm, i didn't know about https://www.mediawiki.org/wiki/Object_cache#Main_stash  . Perhaps MLT should be cached there instead of MainWANObjectCache
[15:52:36] <ebernhardson>	 Behaviour: may involve disk read (1-10ms), semi-persistent, shared between application servers and replicated across data centers.
[15:53:41] <inflatador>	 ask your doctor about Main_stash!
[16:34:43] * ebernhardson wonders how to convince chrome to go to the "right" place when i type phab...it's been trained over years to open https://phabricator.wikimedia.org/tag/discovery-search-sprint/
[16:55:05] <dcausse>	 ebernhardson: re main_stash you're mostly interested in the cross-dc replication?
[16:55:36] <ebernhardson>	 dcausse: also that it stores on-disk instead of in memcached
[16:56:25] <ebernhardson>	 i remember before we considered longer cache times, but didn't want to set it too long because it was going to be 50GB+ that had to be held in memory
[16:58:20] <ebernhardson>	 memcached looks to be 2.7TB in both eqiad and codfw, so it's not like the 50GB is the end of the world, but still seems a bit much. Machines are 128GB memory each which means almost half a server just to cache MLT
[16:59:31] <dcausse>	 makes sense, was never really sure what drives evictions, if it almost always ttl or sometimes capacity
[17:00:32] <ebernhardson>	 I guess my last analysis was https://phabricator.wikimedia.org/T264053#6507373 which is 1 day ttl is ~28.2GB, and 7 days TTL would be ~41G.  If somehow everything was cached it would be ~400GB (sounds dubious?)
[17:02:58] <ebernhardson>	 i have no clue what drives evictions either, and not sure how to find out
[17:03:02] <dcausse>	 everything is all content pages getting a morelike?
[17:03:22] <ebernhardson>	 yea everything is taking the number of docs in *_content and multiplying
[17:03:28] <dcausse>	 ack
[17:04:12] <dcausse>	 the morelike cache does not use getWithSet so we don't get much of the usual metrics
[17:04:22] <ebernhardson>	 i suppose those numbers also assume our cache keys work as expected and there is only 1 key per content page. Hopefully true :)
[17:05:25] <dcausse>	 yes... not sure that's true, mobileapps might possibly ask for more than 3 hits?
[17:05:52] <ebernhardson>	 hmm, i suppose this could be migrated to getWithSet, should be possible
[17:08:49] <dcausse>	 unrelated but should we switch deployment-prep to opensearch?
[17:09:04] <ebernhardson>	 umm, almost certainly yes
[17:09:45] <dcausse>	 ok filing a task, seems odd to move prod before the beta cluster :)
[17:09:56] <ebernhardson>	 yea
[17:12:43] <inflatador>	 oh yeah, I've kinda forgotten about that environment completely ;(
[17:13:31] <dcausse>	 filed T389971
[17:13:31] <stashbot>	 T389971: Migrate deployment-prep elasticsearch cluster to opensearch - https://phabricator.wikimedia.org/T389971
[17:17:15] <ebernhardson>	 re: write isolation, per docs/settings.txt it currently works as intended, with all clusters isolated by default. But i suppose it opens the question, is that an appropriate default?
[17:17:53] <inflatador>	 Who should I talk to before I update anything in deployment-prep?
[17:18:04] <ebernhardson>	 inflatador: thats the magic of deployment-prep, no-one owns it :P
[17:18:17] <dcausse>	 inflatador: it's partly in puppet and partly in horizon :)
[17:18:51] <dcausse>	 ebernhardson: perhaps I'm missing something but I don't think it adds much to have this extra-hop
[17:19:12] <inflatador>	 The magic of un-reproducible environments ;p
[17:19:42] <ebernhardson>	 dcausse: i would generally agree, in particular if there is only a single configured cluster.  I suppose there is a wider question, if WMF is the only place with multiple cirrus clusters, and we do all the writes from SUP now, should all this be simplified away?
[17:21:03] <dcausse>	 ebernhardson: true, does that mean we revert back to classic jobqueue retries and remove ElasticaWrite?
[17:21:23] <dcausse>	 I don't think we support the frozen index bits anymore
[17:21:34] <ebernhardson>	 dcausse: i think we still need ElasticaWrite for retrys, that was it's original intent. But we probably dont need write isolation, job queue partitioning, maybe more
[17:22:48] <ebernhardson>	 CirrusSearchPrivateClusters is also perhaps superseeded by having groups in CirrusSearchWriteClusters now...a few things that are kinda vestigial
[17:23:16] <dcausse>	 ebernhardson: I'm all for simplifying all this
[17:25:03] <ebernhardson>	 I wonder if i remember enough about how it works to simplify without breaking things :P
[17:39:57] <dcausse>	 sigh... https://github.com/opensearch-project/opensearch-build/issues/184
[17:40:30] <dcausse>	 it's solved for opensearch 2 I guess? but it's a bit of maze to find the url of a plugin
[17:40:43] <ebernhardson>	 :S
[17:41:07] <inflatador>	 do y'all consider deployment-prep a blocker for rolling out to production?
[17:41:46] <dcausse>	 inflatador: I'm not sure, we already moved cloudelastic
[17:41:56] <ebernhardson>	 inflatador: in principal, probably? But thats a weak yes.  In theory we want deployment-prep to be able to recreate problems, but in practice we don't use it
[17:42:52] <dcausse>	 inflatador: perhaps we should do it at least before rolling search traffic to opensearch once codfw is done?
[17:43:07] * ebernhardson notes that part of the problem of finding the zips is that the usage of S3 for so many things means simple things like directory listings to browse around and find your .zip don't tend to exist
[17:43:26] <inflatador>	 dcausse ACK, I'm happy to do it first if that gives us more confidence
[17:43:27] <dcausse>	 no directory listing is a huge pain
[17:43:51] <inflatador>	 speaking of S3, that's going to break on deployment-prep for sure
[17:44:04] <dcausse>	 ?
[17:44:41] <dcausse>	 inflatador: you mean s3 secrets?
[17:44:47] <inflatador>	 I mean, the opensearch instances won't start without their S3 config...unless the puppet is different there? I guess I don't know for sure
[17:45:40] <dcausse>	 I don't think there are any s3 repo setup in deployment-prep
[17:47:11] <inflatador>	 yeah...unfortunately the Openstack Ceph is not compatible with Horizon projects that have a hyphen in their name
[17:47:36] <inflatador>	 I can create a user/bucket on the search project, but where to put the secret?
[17:48:32] <ebernhardson>	 i think the theory goes that there are no secrets in deployment-prep? But that could be 10 years old and out-of-date
[17:48:35] <dcausse>	 inflatador: I don't think we need anything related to s3 in deployment-prep 
[17:49:02] <dcausse>	 the plugin requires a default set of secrets?
[17:49:44] <inflatador>	 dcausse not sure yet...I was just thinking that if it applies the cirrus::opensearch role it would need something
[17:49:58] <inflatador>	 but that could be 100% wrong. Let me do some digging and I'll get back to y'all
[17:50:01] <ebernhardson>	 inflatador: i would put the string `not-really-a-secret` and call it a day :P
[17:50:39] <inflatador>	 oh yeah, I was more concerned that the plugin would try to connect to the S3 endpoint and refuse to start if it wasn't there
[17:50:58] <ebernhardson>	 hmm, i suppose my hope would be that it doesn't do anything until the snapshots api is invoked, but not really sure
[17:50:59] <inflatador>	 But I'm piling assumptions on top of assumptions at this point ;P
[17:51:06] <ebernhardson>	 we all do :)
[17:51:10] <ebernhardson>	 it's the only way to survive
[17:51:25] <dcausse>	 :)
[17:52:25] <inflatador>	 No worries, will dig thru deployment-prep horizon once I get back from lunch
[18:08:25] <dcausse>	 dinner
[18:24:38] <inflatador>	 back
[18:29:30] <inflatador>	 DPE SRE has a flapping alert for `AirflowDeploymentUnavailable: airflow-scheduler.airflow-search is unavailable on dse-k8s-eqiad ` ... any idea what might be causing that?
[18:30:56] <inflatador>	 Ben and I are looking, so NBD if not
[18:31:09] <ebernhardson>	 i have some vague memory that it occured on another teams thing too, but i'm forgetting the name
[18:31:52] <ebernhardson>	 https://wikimedia.slack.com/archives/C055QGPTC69/p1742325253186869 is what i was thinking of
[18:32:09] <ebernhardson>	 but i guess thats slightly different
[18:32:19] <ebernhardson>	 in general, i don't know how the new k8s variant works :P
[18:38:16] * inflatador is learning now ;P
[18:47:43] <inflatador>	 we've just increased scheduler resources, exactly as in your link :)
[18:48:35] <inflatador>	 spoke too soon...we're still getting 503s from teh scheduler
[18:48:42] <ebernhardson>	 :(
[18:53:13] <gmodena>	 oof
[18:53:54] <gmodena>	 i wanted to release mjolnir 2.5.1, but I'm afraid ci already bumpted to 2.6.0.dev https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/blob/main/.bumpversion.cfg?ref_type=heads
[18:54:20] * gmodena wonders how to trigger major and patch releases with wmf_workflow_utils
[18:54:32] <ebernhardson>	 gmodena: there is a button on the pipelines page, sec
[18:55:07] <ebernhardson>	 gmodena: the play icon to the right that opens a dropdown, on commits to main, has a 'trigger_release' option
[18:55:16] <ebernhardson>	 https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/pipelines
[18:55:37] <ebernhardson>	 i think it might not be available until the CI finishes after the merge
[18:56:08] <gmodena>	 ebernhardson ack on that. But do you think setting a VERSION variable in there would work?
[18:56:09] <ebernhardson>	 i think we have that same thing in most of our repos, i find it easier than pushing a tag (although pushing a tag is of course pretty easy)
[18:56:34] <ebernhardson>	 gmodena: hmm, no clue. I've always just let it version things. I know it's not semver, but personally i could have dates as the version :P It's not like we have external users
[18:56:34] <gmodena>	 i would like to release the current main, with the MR you accepted (thanks!), as 2.5.1
[18:56:40] <gmodena>	 but I think I've missed the boat
[18:57:05] <gmodena>	 ebernhardson eh
[18:57:19] <gmodena>	 ok with you if I release as 2.6.0 then?
[18:57:22] <ebernhardson>	 ea
[18:57:24] <ebernhardson>	 yea
[18:57:37] <gmodena>	 feels a bit silly, but i don't want to manually mess with tags / file configs
[18:57:49] <gmodena>	 ebernhardson thanks
[18:57:50] <ebernhardson>	 feels totally normal to me :) Each release gets a +1
[18:58:03] <gmodena>	 ok!
[19:18:05] <inflatador>	 Re: Airflow deployment alerts, it was actually due to a misbehaving ceph node. I'm writing some docs to connect the dots
[19:26:36] <inflatador>	 OK, extremely rough writeup here, but the TLDR is that some pod crashes can be traced to misbehaving ceph volumes https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Ceph/Troubleshooting
[19:29:33] <ebernhardson>	 makes me more worried about the elastic on ceph ideas :P
[19:30:54] * ebernhardson is going to have a hard time writing not-elastic :P
[19:32:45] <inflatador>	 agreed on both points ;) . But I feel like these are growing pains...pretty much everyone else in the world is using crazy hyperconverged storage. Hopefully we can work the kinks out
[19:33:48] <ebernhardson>	 i suppose my worry is that other places also have networks at a scale thats 10x what we do
[19:35:27] <ebernhardson>	 like, i've read that many places are running 50GBE within racks, and 25GBE between racks.  With 100GBE used in ML dedicated clusters
[19:36:22] <ebernhardson>	 some quick searching suggests 200GBE and 400GBE are not unheard of for between racks 
[19:37:36] <inflatador>	 I dunno...I think we need some work on QoS, but we had entire DBaaS products running on bonded 10 Gbps connections between racks
[19:38:06] <inflatador>	 10 Gbps or even 1 Gbps between racks. Granted, that was built 10 years ago and I haven't seen it in 5 years
[19:39:01] <inflatador>	 sorry 1 or 10 Gbps within racks, between 10 and 20 Gbps between racks
[19:41:39] <ryankemper>	 Backport deploy in 20 minutes, after which we'll have mediawiki querying categories from `wdqs-main` instead of `wdqs-full`
[19:41:50] <inflatador>	 {◕ ◡ ◕}
[19:42:05] <ebernhardson>	 i dunno, it's certainly possible, i suppose i just worry we never built the DC networks out for high-bandwidth applications.   But will see how it works
[19:43:05] <inflatador>	 yeah, I think we'll need to do QoS, bonding, maybe beefier network gear like you're saying
[20:24:57] <gmodena>	 File does not exist: hdfs:/wmf/cache/artifacts/airflow/search/mjolnir-2.6.0.conda.tgz
[20:24:58] <gmodena>	 mmm
[20:27:52] <ebernhardson>	 hmm, that is odd. Usually the artifact handling just works™
[20:28:51] <ebernhardson>	 i think it's done as part of `scap deploy` from deployment host?
[20:29:05] <ebernhardson>	 (not sure if thats even a thing in the new k8s stuff)
[20:30:18] <gmodena>	 i think we still need to scap
[20:33:09] <ryankemper>	 Thoughts on how to test https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1124535/ ? The scap deploy is finishing up now
[20:33:27] <ryankemper>	 Probably exec into mediawiki pod and try to curl? Although curl might not be installed on the container
[20:33:40] <ryankemper>	 Can also look at https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m&var-cluster_name=wdqs-internal-main&var-graph_type=%289103%7C9194%29 although the qps graph is hard to read
[20:33:43] <ebernhardson>	 ryankemper: issue some deepcat queries, see if it doesn't fail
[20:33:51] <gmodena>	 https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes#DAGs_deployment does not mention scap. I'll ping the broader group in slack
[20:34:07] <inflatador>	 you could check the access logs on the internal-main hosts too, see if stuff is showing up there
[20:34:13] <ebernhardson>	 ryankemper: https://www.mediawiki.org/wiki/Help:CirrusSearch#Deepcategory
[20:34:49] <ebernhardson>	 ryankemper: essentially, use the browser plugin to route requests to the appropriate mwdebug host, then issue a deepcat query like the example
[20:36:28] <ebernhardson>	 curious that 'artifact' doesn't show up on the k8s or airflow pages anywhere :S
[20:36:33] <ebernhardson>	 (on wikitech)
[20:37:12] <ebernhardson>	 and https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide still references scap :P
[20:38:55] <ryankemper>	 ebernhardson: so if I got appropriate results for `deepcat:"musicals"` on `en.wikipedia.org` with `X-Wikimedia-Debug:backend=k8s-mwdebug` via the browser tool then I should be good?
[20:39:21] <ebernhardson>	 ryankemper: assuming scap is at the point where it tells you to it's been deployed to mwdebug and is waiting for confirmation that its ok, yea that should be it
[20:39:26] <ryankemper>	 yeah it is
[20:39:29] <ryankemper>	 great
[20:39:32] <inflatador>	 ryankemper LGTM assuming the above as well
[20:42:36] * ebernhardson takes minor offence to the statement (in a commit message) "The default CirrusSearch configuration does not work out of the box".  The problem is they don't have elasticsearch installed and are complaining that it gives errors :P
[20:42:49] <ebernhardson>	 imo thats what cirrus should do
[20:43:15] <inflatador>	 Yeah, I dunno what the alternative would be
[20:43:40] <ebernhardson>	 they asked for the default to be setup so cirrussearch doesn't attempt to write anywhere when installed unless otherwise configured
[20:44:49] <ebernhardson>	 it's just a weird CI setup where they have Cirrus because of extension dependencies, but they don't actually want to use cirrus
[20:45:04] <inflatador>	 So I guess they just want a silent failure when it doesn't exist or something?
[20:45:09] <inflatador>	 maybe a warning?
[20:45:30] <ebernhardson>	 i gave them some config settings that match prod, where cirrus doesn't attempt to write (and assumes some external application will do that)
[20:48:02] <inflatador>	 interesting, I guess that's how it works now due to the new SUP?
[20:48:28] <ebernhardson>	 yes for SUP we added a way to disable direct writes 
[20:49:27] <gmodena>	 welp. discolytics is not happy with the query_clicks_ltr  schema.
[20:49:35] * ebernhardson is mostly commenting here to avoid complaining on the patch...they can do what they want i guess :P
[20:49:46] <ebernhardson>	 :S
[20:52:29] <ebernhardson>	 gmodena: the `raise ValueError('schema does not match any known datetime partitioning')` error?
[20:52:54] <gmodena>	 yes - I'm just checking what's up
[20:53:35] <ebernhardson>	 hmm, query_clicks_ltr is partitioned by year/month/day. should work :S
[20:53:42] <ebernhardson>	 err, query_clicks_daily
[20:54:48] <ebernhardson>	 that error message should probably include the table name just to make things more obvious
[20:54:58] <ebernhardson>	 i guess the function doesn't have the name though
[20:56:37] <ebernhardson>	 did i do something silly like drop those columns earlier?
[20:57:51] <ebernhardson>	 gmodena: sigh, yes the problem is mjolnir.cli.helpers in the `require_daily_input_table` it drops year/month/day.  I suppose the proper integration would move that condition into the input param helper
[20:58:32] <gmodena>	 ah!
[20:58:35] <gmodena>	 good catch
[20:59:52] <ebernhardson>	 i suppose that would also move the start/end date arguments into the helper as well
[21:01:22] <ebernhardson>	 or alternatively, i suppose a more complete integration would be to delete the HivePartition class from mjolnir and migrate everything to the HivePartition class in discolytics (it started in mjolnir, then we moved it but never got around to using discolytics in mjolnir)
[21:02:00] <ebernhardson>	 i should probably have remembered all this last week :(
[21:02:24] <gmodena>	 np
[21:02:34] <gmodena>	 i should have tested better
[21:02:36] <ryankemper>	 inflatador: anything in mind for pairing today?
[21:02:43] <gmodena>	 and not just believe integration tests :)
[21:02:53] <gmodena>	 i'll sleep on it and see how to refactor tomorrow morning 
[21:02:53] <ebernhardson>	 integration testing spark has always been hard
[21:03:01] <ebernhardson>	 and slow...
[21:03:01] <gmodena>	 yeah, airflow too
[21:03:31] <gmodena>	 just spotted a typo (variable name) in the MR i just merged. Sigh. Will do a quick f/up on that too.
[21:03:42] <inflatador>	 ryankemper was gonna look at T389971
[21:03:43] <stashbot>	 T389971: Migrate deployment-prep elasticsearch cluster to opensearch - https://phabricator.wikimedia.org/T389971
[21:04:35] <gmodena>	 ebernhardson for the sake of wrapping up this data retention bit of work, I'd be keen in refactoring filtering logic to the helpers class
[21:05:02] <gmodena>	 and eventually refactor everything to discolytics at a later stage (whch I won't get to this week :()
[21:05:05] <gmodena>	 what do you think?
[21:05:07] <ebernhardson>	 gmodena: agreed, the simple answer for now is fine. Maybe maybe a ticket for a followup to use HivePartition from discolytics
[21:05:15] <ebernhardson>	 across the whole package
[21:05:18] <gmodena>	 ack
[21:26:21] <gmodena>	 i published a couple more examples of llm output at https://people.wikimedia.org/~gmodena/search/T388549/llm-judge/ 
[21:31:16] <gmodena>	 for now I've only been using random pages (this is just a sample), holler if there's some query you'd be interested to try! cc / Trey314159 ;)
[21:31:24] <gmodena>	 does not have to be only on enwiki
[21:38:39] <ebernhardson>	 gmodena: i might suggest trying a few with only the top 3, since thats what the UI will show
[21:39:15] <ebernhardson>	 ahh i see cycling has the top 3
[21:40:34] <gmodena>	 ebernhardson ack. dcausse made the same point :D
[21:40:47] <gmodena>	 here went for 10 to see if more context might help
[21:40:52] <gmodena>	 but I don't think it does
[21:41:02] <gmodena>	 (based on the few experiments)
[21:42:24] <gmodena>	 welp. It got late here. Talk to you tomorrow!
[21:42:27] * gmodena waves