[08:16:59] streaming updater failures yesterday were caused by etcd having troubles it seems [08:46:52] inflatador: (for when around) is there anything left to do in T298525 ? Or should you move it to "needs review" ? [08:46:52] T298525: Tune "BlazegraphFreeAllocatorsDecreasingRapidly" alerts - https://phabricator.wikimedia.org/T298525 [10:11:07] hi folks, I'm looking at removing some tech debt in how we do network probes for services in T291946 and noticed the search-https certificates don't include the .discovery.wmnet record in SAN. I'm assuming this is for historical reasons, any objections to add search.discovery.wmnet there ? [10:11:07] T291946: Move service::catalog checks (“monitoring” section) to blackbox exporter and Alertmanager - https://phabricator.wikimedia.org/T291946 [10:11:50] godog: no objection! [10:12:35] The SSL certs for search were created a long, long time ago. I think we've improved the way we're managing internal certs since then, but I have not followed those evolutions. [10:12:48] gehel: thanks! I'm opening a separate task to track that, since I think it makes sense to move the certs to cergen too [10:12:51] yeah exactly [10:12:56] inflatador and ryankemper should probably have a look at cleaning up this [10:13:14] godog: Thanks for creating a task! [10:13:18] I'll CC them on the task [10:19:16] {{done}} T299633 [10:19:16] T299633: Add search.discovery.wmnet to search certs SAN - https://phabricator.wikimedia.org/T299633 [10:24:13] godog: thanks again! [10:25:38] sure! [10:37:14] errand + lunch [11:18:19] Another search query, and im tyring to do possibly evil things, so if its silly please let me know. I'm wondering why "*5x8*" might get a result, but "P109=*5x8*" does not, but "P109=YEAMK is a 5x8*" does ? [11:21:57] addshore: properties we index are not "tokenized", the idea is that we considered them to be ID, codes or other things that may not be searched partially [11:23:10] the reason they appear when searching without the keyword is that I think we copy this content to the main search field which is tokenized [11:23:42] gotcha [11:31:18] We wonder if there's a way to have combined queries that operate on a single field, i.e. something like "prefix must be P109= and this same field must also match *banana*" [11:31:20] but guessing no [11:43:49] addshore: sadly no, hadwbstatement has been made to find structured data search not for natural language text/search [11:44:54] and this also applies if we talk directly to elastic? [11:45:33] yes because we don't have an elastic field per property but a single field where the property id is encoded into the data [11:47:10] {doc: {"statement" : [ "P180=Q123", "P109=random text" ]}} and not {doc: { "P109": "random text", "P180": "Q123" }} [11:48:19] addshore: you could use expensive wildcard queries tho [11:51:41] this would look like: {"wildcard": {"statement_keywords": {"value": "P109=*banana*"}}} [11:52:16] there are limitations tho, https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html [11:53:12] lunch [13:03:53] dcausse: when you are back [13:06:32] thanks for the hints dcausse , looks like we are making ome headway with our hack :D [13:10:39] ejoseph: I'm around now, sorry about that [13:37:41] gehel: when you have time: https://gerrit.wikimedia.org/r/c/operations/puppet/+/755706 [13:38:23] Looking [13:40:00] ejoseph: can you jump in a meet to confirm your ssh key? [13:40:13] meet.google.com/mgk-egby-ipf [13:43:37] merged, it will take up to 30 minutes to propagate on all servers [13:45:13] gehel: thanks! [14:10:36] Greetings all [14:17:05] o/ [14:59:02] errand [15:11:41] setting up my work laptop, so may be a bit [15:20:03] gehel: Should I move the ORES ticket to needs reporting? Or should it stay open for more work in future? [15:20:36] Let's move it to needs reporting. Given the discussion today, I don't think we'll do additional work on this any time soon [15:20:55] We can always re-open it later, or create a new (and more specific) one in the future [15:21:38] Sure, doing that. Thanks! [15:28:49] dcausse: can we continue [15:29:26] ejoseph: ok [15:29:31] joining the same meet [15:59:44] huh, a certain grant is apparently getting back to his roots, and posted an issue to the es-ltr plugin :) [16:01:06] cool! [16:02:31] mpham, ejoseph: retrospective: https://meet.google.com/ssh-zegc-cyw [16:57:45] https://en.wikipedia.org/wiki/Sp%C3%A4tzle [17:00:27] workout, back in ~30 [17:46:09] and back [18:08:53] ryankemper or anyone else, this is the puppet course I bought https://www.udemy.com/course/learn-puppet/ . I just started so I can't really say whether it's good or not, but the price is right ;) [18:27:20] * ebernhardson ponders the comments earlier about a bash script where it doesn't belong, while looking at a bash script that writes a bash script over ssh that will write a third bash script [18:27:45] (at least it's over vagrant ssh to a local container) [18:47:43] lunch, back in ~30 [18:49:01] dinner [19:07:03] back [19:26:02] ryankemper, inflatador : I'll be 5' late, still need to do the injection with Oscar [19:26:11] gehel: ack [19:27:52] ACK [19:33:34] and back [19:33:41] ryankemper: ^ [19:43:25] lunch [20:02:11] Hi search team, us on the data engineering team are evaluating a data catalog that has a elasticsearch dependency; if you were setting up a new cluster, what version and what distribution (elasticsearch or opensearch) would you choose? [20:07:15] razzi: per licensing likely only opensearch has a future here, i'd lean towards the latest opensearch [20:08:30] ty ebernhardson ! [20:25:44] gehel: inflatador : Ah I figured out what my subconscious confusion was on the lvs restarts...I'd thought (before looking today) that `wcqs` was all the way back in `service_setup` (the initial stage), but we're in `lvs_setup` (the second stage), and the docs say to do the restart after moving to `lvs_setup`. So the TL;DR is I think that means we don't actually need to do another pybal restart [20:25:50] I'll need to read the docs more tho to confirm [20:26:31] https://wikitech.wikimedia.org/wiki/LVS#Configure_the_load_balancers is where it describes the pybal restart steps which occur *after* the service is moved into `lvs_setup` [20:27:07] Ah and yeah looking at the ticket history I see that on dec 1 I moved it back into `lvs_setup` and did the restarts (I just didn't remember), so stuff is making sense now: https://phabricator.wikimedia.org/T280001#7542629 [20:27:34] inflatador: so instead of pybal restarts, when we pair in an hour let's work on getting the service from `lvs_setup` into `monitoring_setup` [20:28:17] inflatador: if you want to read a bit in advance, https://wikitech.wikimedia.org/wiki/LVS#Add_monitoring_to_the_load-balanced_services covers the `monitoring_setup` steps. we'll need to do that DNS stuff basically [20:30:59] ryankemper ACK, will check it out. Setting up my new laptop, so may be AFK for a bit [20:31:16] lunchtime for me [20:32:23] ooh, you get the new 32G spec laptop? We just upgraded default eng laptop and i'm a little jealous :) [21:03:11] ebernhardson 64GB M1 Max [21:03:11] M1: MediaWiki Userpage - https://phabricator.wikimedia.org/M1 [21:03:20] even worse :P [21:03:36] I love how that randomly triggered a bot [21:03:57] yea phab tags do things, T295734 it's nice sometimes [21:03:58] T295734: Bring up two copies of the CirrusSearch browser integration env in cloud - https://phabricator.wikimedia.org/T295734 [21:04:26] does it do pates to? P18948 [21:04:33] apparently no pates :) [21:04:38] pastes, wow i cant type [21:58:20] must be a bit rusty, can't think of why we use /[?*]/u as the regex to decide between intitle doing plain or stemmed queries. But we've used it since at least 2015 so it probably works? [22:26:13] and the answer is...5f0267a2a16. It implements wildcard queries, not plain for stemmed queries. it just happens to be there. stemmed is perhaps indirectly implemented via query_string query but not completely clear [22:48:21] looks like cindy gave a V+1 running elastic 6.8 with ejosephs new package. cool! [22:55:49] \o/ [23:11:59] until tomorrow [23:12:04]