[08:41:27] Trey314159: nice writing! Another blog post coming up? [09:57:59] lunch [12:24:18] gehel: Maybe! I'll see if Diff is interested [13:21:01] gehel per conversation with btullis , Airflow has been de-prioritized ATM. Anything you'd like me to focus on? I was going to start on T362105 (mutual Opensearch) but could work on T346189 or anything else you have in mind [13:47:47] T362105: EPIC: Mutualized opensearch cluster - https://phabricator.wikimedia.org/T362105 [13:47:48] T346189: Migrate search-loader into Kubernetes - https://phabricator.wikimedia.org/T346189 [13:56:45] o/ [14:06:52] \o [14:12:45] o/ [15:02:20] search triage: https://meet.google.com/eki-rafx-cxi (cc: ryankemper, pfischer) [15:02:49] huh, apparently 60TB SSD's are a thing: https://www.solidigm.com/products/data-center/d5/p5336.html [15:47:25] Trey314159: if anyone at Diff tells you that topic is too technical, debt is the new manager of techblog and can help you publish there. https://phabricator.wikimedia.org/project/board/4654/ [15:50:51] bd808: I didn't initially think of the Tech Blog because it was on hiatus or something the last time I wanted to publish a blog.. but this morning I remembered it existed and it seems to be updating, and I think it might be a better venue anyway—though I'm curious if anyone has a strong opinion about Diff vs Tech Blog as a venue. [15:52:16] Trey314159: there was going to be a project to figure out if techblog could just merge into Diff at this point, but I think it got lost in the team shuffles last year. [15:53:02] The historical rise of techblog this last incarnation was because the blog that Comms ran did not want "deep technical" content. [15:54:03] Interesting. I'll talk to Deb and then work on getting it out there somewhere. [15:54:31] awesome. [16:09:34] o/ random question: are redlinks indexed at all in the Search indexes for Wikipedia? I assume not because no text to match them on except the title and you can use Wikidata labels instead for surfacing content outside of a given language edition but figured I'd check. context: I'm wondering whether it's possible to apply tags to redlinks in a way that they'd be discoverable via the Search API as candidate articles for translation. [16:15:18] isaacj: nope, the search indexes are per-page and we only index pages that exist. Redlinks are unrepresented [16:15:32] if there were use cases, we could perhaps add a field to pages that list all the redlinks or something like that [16:15:54] the redlinks within that page [16:16:24] but that would probably have annoying data quality issues, pages would be created but updating the search engine to remove redlinks in all those places would be hard to get right [16:56:17] dinner [16:57:00] it looks like next steps for federation is to to build and deploy the rdf repository? poking over it [16:59:18] dcausse looking at your alerts patch, how long did you say it would take for the sanitizer metrics to show up in prometheus? [17:02:04] ebernhardson: thanks for the details. that makes sense. i don't think it's worth trying to put together a hack at this point and i'm hopeful we can come up with a better solution. the issue at play is that we'll eventually want to enable editors to compile translation lists (articles to create in their wiki) and have these be exposed as filters in our recommender systems. for example, maybe it's a list of women scientists that don't [17:02:04] yet have articles in spanish wikipedia. and then an editor comes in and they want to find articles from that list and further filter down to women scientists from Ecuador (either via country tags which aren't available yet but I'm working on that as an extension of articletopic or via something like `morelikethis:Ecuador`). the challenge is that those filters wouldn't work on Spanish Wikipedia because the articles don't exist yet [17:02:04] there and would only work on the "source" wikis where the articles do exist and have tags/text available. one approach would be indexing these lists not as the desired articles (redlinks) but as the possible source articles to translate over across the various wikis. then when an editor comes in, they select their desired source language and we can run a query to grab the list of available articles for translation on that wiki and [17:02:04] apply filters as needed. that would work but either constrains the organizers to identifying the specific sources to translate from or requires some messy update pipelines to propagate the translation lists across the different language editions. so i'm trying to think through alternatives that might not be quite as prone to breaking while still preserving some flexibility. nothing urgent at the moment just trying to get ahead of the [17:02:04] problem when it arises. [17:03:52] inflatador: line 34 uses [1w] for the aggregation, and metrics started coming in friday. So probably next week would work. re: https://gerrit.wikimedia.org/r/c/operations/alerts/+/1054317/2/team-search-platform/cirrussearch_k8s.yaml [17:05:41] isaacj: hmm, maybe we can do something but i'll have to think on that for a moment :) [17:07:12] ebernhardson ACK, I tried using [1d] in Grafana explorer but didn't get any results, maybe I PEBKAC'd [17:09:17] :thumbs up: thanks let me know if you think of anything and I'll continue to brainstorm too [17:13:39] inflatador: hmm, 1d should be there. if not we are more issues to solve :S looking [17:15:10] inflatador: suspect you need to drop `> 100000` from the end, that part triggers the alert and everything should ideally be under it [17:16:28] ebernhardson I did get rid of that, but I was getting nothing. I say "was" because it seems to be working fine now. Maybe I was using the wrong prometheus instance or something [17:16:42] inflatador: ahh, maybe. I tested against thanos and it pulled something up [17:19:33] yeah, I think that was it. The other metric we're using ( mediawiki_CirrusSearch_pool_counter_seconds_count) exists in both k8s and ops prom instances, but it looks like the data only exists in k8s [18:12:49] lunch, back in ~50 [18:32:32] hmm, 404 not found at https://archiva.wikimedia.org/ . Expected change? [18:32:46] i have some vague memory of a security problem [18:33:18] but not finding anything obvious in puppet [18:42:30] (it's not necessary, at least for wdqs, but still curious) [18:45:22] put up the patch to deploy 0.3.144 of wdqs, but it's been awhile since i've deployed wdqs and don't know i can really verify it's not-broken. Patchlist for the deploy looks reasonably simple though, shouldn't be too risky [18:51:45] back [18:53:31] ryankemper if you're around today, maybe we can deploy wdqs per ^ ? [19:02:36] inflatador: yeah that sounds good [19:03:01] inflatador: our normal pairing time isn’t great for me but I can pair in a couple mins if we want to roll that now [19:03:49] ryankemper sure, I can do now...have a hard stop in 45m though [19:07:01] inflatador: that's fine I can babysit the rest of the deploy [19:07:07] inflatador: grabbing water then joining meet [19:07:17] ryankemper ACK, sounds good [19:41:56] picking up kids, back in ~30 [20:39:33] back...and wow, that took a lot longer than I thought [20:46:42] i went to in-n-out last night. 4 minute drive away. Took an hour :S Different but maybe similar feeling :) [20:53:30] similar, except I got a carful of kids and cellos instead of food ;(