[06:57:32] ejoseph: as far as your production access (not cloud) is concerned, we need to fill out this access request: https://phabricator.wikimedia.org/T294379 (instructions follow) [06:57:42] I've filled out some basic parts, but you'll need to edit the ticket body to add in (1) your Wikitech username, (2) your ssh public key (contents of `~/.ssh/wmf_prod_ed25519.pub`), and (3) read and sign https://phabricator.wikimedia.org/L3 and update the corresponding TODO in the ticket body to mention that you've filled it out [06:58:03] All 3 of those things have TODOs in the ticket body so it should hopefully be pretty clear which lines to change [08:01:42] wdqs1013 acting up again ... [08:01:58] ryankemper: I’ll do that now [08:01:59] and started to at almost exactly the same time as the last [08:02:32] I'm doing the restart [08:07:30] thanks! [08:09:41] we have an alert for processing latency, maybe we should have one for lag as well, since it's our SLO [08:09:49] or we do and it didn't fire? [08:10:39] we have an alert on lag already but it fires at like 10hours [08:11:28] plus it only fires if the service properly reports the metric which is not the case here [08:11:35] yep [08:12:01] but yes if we have an SLO alerts should setup accordingly [08:12:04] I'm guessing this should fire a completely different set of alerts [08:12:38] the metric should be computed from the updater which is more stable [08:13:00] consumer you mean? [08:13:05] yes [08:14:10] currently it reports the lag in millisec, but should report the time so that it's up to prometheus to do the now() - update_time operation, preventing the series to stall [08:14:33] makes sense [08:14:52] it's how it's done for blazegraph [08:15:29] yes, but the prometheus exporter is more likely to stay alive and functional on the consumer [08:15:45] true [08:16:34] thinking about it it should be a couple lines to add to the code [08:16:52] we should use that for SLO calculation as well - using blazegraph means that we're being a bit dishonest about the calculation [08:18:05] yes we discussed that yesterday, it's true it feels like we take benefit from blazegraph being down to make our SLO happy [08:18:43] I'm all for having clearer metrics that are less prone to confusion [08:18:44] well, I could spin that in a way it makes sense, but it's not really what we agreed on [08:19:30] we could have both during a transition period [08:21:28] saying that the previous system was unable to report lag properly in this situation so we use SLOv1 for comparison in the short term but we aim at having a SLOv2 which is more honnest [08:31:20] makes sense [08:51:35] FYI - I addressed all the comments on the technical communication for streaming updater, if there are no more, I'll be posting it in few hours to wikidata-tech-l (should I go with wikidata-l as well?). [08:55:50] no clue, it's fairly technical so -tech-l makes a lot of sense but to be sure I'd ask either Mohammed, Luca (Sannita) or Léa (probably via #wdqs on slack) [08:56:29] will do, thx for the advice [09:00:05] ejoseph: I'll be a few minutes late for our meeting [09:00:39] gehel: ok, just ping me when you are back [09:03:32] ejoseph: interview ran late, and I need a 5 minutes break [09:03:35] * gehel is back in 5' [09:28:10] tanny411: if you haven't met with Emmanuel, you probably should! He is the newest member of the team. I'm sending him your way to schedule some time with you (cc ejoseph ) [09:28:56] gehel: sure! [09:28:57] ejo [09:29:05] ejoseph: nice to meet you! [09:33:34] tanny411: I've lost you :/ waiting for you to reconnect [09:34:36] connecting... [09:38:39] Damn, I've lost you again :/ [09:47:55] lunch [10:17:11] lunch [10:23:52] I just got delivered something I definitely shouldn't eat on my diet :D [10:24:22] I'll wait with the consumption until tomorrow's visit to dietician (and to celebration of course) [10:33:42] tanny411: Hi, nice to meet you too [10:44:01] ejoseph: feel free to set up a call next week. we can talk! I'm on UTC+6 :) [10:58:15] Ok cool [13:04:34] break [13:34:36] Launch [15:03:48] dcausse: mtg [15:03:54] Wednesday meeting has started: https://meet.google.com/yau-mkip-tqg [15:03:58] oops [15:04:02] cc: ryankemper, dcausse, ejoseph [15:04:07] And Connie has joined us! [19:59:31] I wish to upgrade my membership in the dental work club [20:00:20] it turns out that the tooth with the crown is not the root cause of my problems... the one next to it also needs a root canal and crown. and then after they must go back to the first tooth and re-do the root canal there. [20:23:41] ouch, the continuing saga's :) [21:08:31] tltaylor: Redoing a root canal? That's a fairly uncommon achievement! I thought I was the only one... [21:13:17] mpham: i was just checking, turns out we should be targeting elastic 7.10: https://www.elastic.co/pricing/faq/licensing [21:13:43] ah, thanks ebernhardson . will update [21:23:13] a more mundane question: how come the mw api query action doesn't return a relevancy score? [21:28:22] tltaylor: The argument against adding it is usually that it's a meaningless number. The only built in meaning is that bigger numbers are better, and that's already represented in the result list [21:28:32] that's true [21:28:42] but it's preventing me from doing my hackathon idea [21:28:49] tltaylor: if you are just curious, we have debug output. esc [21:28:50] sec [21:29:32] I want a federated search across office, mw.org, meta, and wikitech [21:30:21] tltaylor: in that case, use https://wikitech.wikimedia.org/wiki/Help:CirrusSearch_elasticsearch_replicas [21:30:32] tltaylor: oh, actually that wont get office though [21:31:14] tltaylor: for hackathon level output, you can likely use cirrusDumpResult=1, which gets the raw elasticsearch response https://en.wikipedia.org/w/index.php?search=example&title=Special:Search&profile=default&fulltext=1&cirrusDumpResult=1 [21:31:28] interesting [21:31:30] thank you [21:31:34] if more curious about scoring, there is an explain option too: https://en.wikipedia.org/w/index.php?search=example&title=Special:Search&profile=default&fulltext=1&cirrusDumpResult=1&cirrusExplain=pretty