[00:17:41] 10Lift-Wing, 10Patch-For-Review: Factor out feature retrieve functionality to a transformer - https://phabricator.wikimedia.org/T294419 (10ACraze) I added a CR with my WIP revscoring transformer that I started for articlequality a while back. It needs a bit more work, but it seems doable for this class as it i... [07:19:09] good morning! [07:19:58] I quickly check what we mean with the metrics "score errors" and I am a bit confused [07:21:07] I thought that all those errors corresponded to a similar amount of HTTP responses with non-2XX return codes, but from the system logs on a couple of ores nodes this seems not the case [07:21:37] for example, I see https://github.com/wikimedia/ores/blob/master/ores/scoring_systems/scoring_system.py [07:22:58] https://github.com/wikimedia/ores/blob/master/ores/scoring_systems/scoring_system.py#L130-L134 [07:23:47] do we know what is the source of errors? [07:24:42] I am worried about lift wing metrics, since we decided (for the moment) to rely on the HTTP return codes of the KServer HTTP server, but is it a safe bet? [07:33:39] https://grafana.wikimedia.org/d/VTCkm29Wz/envoy-telemetry?orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus%2Fops&var-origin=ores&var-origin_instance=All&var-destination=All is for the envoy's proxy used by ORES [07:33:57] and I don't see troubles in there (at least matching with scores errored) [07:37:08] Amir1: o/ (when you have time) - do you know by any chance what an score_errored datapoint corresponds to? What I mean is where I should look to see what's wrong etc.. [11:20:14] elukey: morning :D, I don't know it on top of my head but I can double check and let you know [11:21:59] Amir1: o/ don't worry I'll try to figure it out, it was only if you knew from direct experience, don't spent time on it :) [11:23:19] elukey: I'm slowly waking up. I think it's for cases scoring and edit errors (duh) for things like time out, trying to run a json model (wikidata) on wikitext, etc. etc. [11:33:17] Amir1: ack, will investigate :) [14:56:36] 10Machine-Learning-Team, 10drafttopic-modeling, 10Discovery-Search (Current work), 10Epic: [Epic] Add drafttopic predictions to ElasticSearch index for the Draft namespace where available - https://phabricator.wikimedia.org/T249341 (10Gehel) 05Open→03Resolved a:03Gehel [16:00:50] o/ [16:01:41] elukey: thx 4 the review on the model upload script - i should have been using the `error_code` var, will fix it up today :) [16:06:29] accraze: ahhh okok now it makes sense, I was puzzled while drinking coffee but I wasn't sure :) [16:07:40] accraze: I am wondering now if we should add the script to puppet, instead of our repo [16:08:15] it will likely be used only on stat100x boxes for the moment, where we'll deploy credentials [16:08:20] yeah i was wondering the same yesterday, not really sure where it belongs [16:08:46] I'd vote for puppet since we'd have a way to inject values like the .cfg containing credentials [16:09:38] if you want I can draft what I have in mind for puppet, and you can takeover [16:09:51] yeah puppet makes sense, i'm down! [16:22:05] ack prepping one [17:00:30] accraze: https://gerrit.wikimedia.org/r/c/operations/puppet/+/736848 [17:00:35] this is the initial idea [17:00:45] name of the profile etc.. may change [17:00:51] I'll ask to DE a review [17:00:56] lemme know what you think about it [17:01:07] (we can go through it via meet if you want) [17:11:29] nice! yeah it would be helpful to go through, im not super familiar w/ puppet [17:18:45] accraze: meet.google.com/goq-ycoo-hmn (if you have time) [18:23:41] * elukey afk! o/ [23:16:35] hmmm been thinking alot about T294419 [23:16:58] (revscoring feature transformer) [23:17:50] starting to think the actual transformer image will become quite large (1GB+) [23:19:10] not sure how scalable a generic transformer will be for all the revscoring model classes since both the transformer and the custom model container will need the full revscoring install (and dictionaries etc) [23:19:37] we might just need to have specified transformers for each model class