[08:08:06] hello folks [08:08:12] deploying the new Docker images [08:08:40] the remaining logspam from istio-proxy's envoy should also go down to zero (hopefully) [09:30:19] all new docker images rolled out [09:30:27] the logspam issue is still present though, weird [09:30:36] the fix wasn't enough probably, sigh [09:34:53] (03PS1) 10AikoChou: draftquality: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842358 (https://phabricator.wikimedia.org/T318932) [09:36:00] Morning! [09:38:10] morning! [09:38:22] of course I forgot one error handling in fetch_features [09:38:23] revscoring.errors.UnexpectedContentType: UnexpectedContentType: Expected content of type JSON, but the following can't be parsed (max 50 chars showed): {{Constraint violations report|date=2022-10-09T1 [09:38:35] this is mostly for wikidata, we patched it for ORES [09:38:37] sigh [09:38:40] * elukey writes a patch [09:38:51] I'll review if you send me the link :) [09:45:14] (03PS1) 10AikoChou: topic: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842360 (https://phabricator.wikimedia.org/T318932) [09:46:28] (03PS1) 10Elukey: extractor_utils: handle more exception cases [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842361 (https://phabricator.wikimedia.org/T320374) [09:46:34] there you go --^ :) [09:47:43] (03PS1) 10AikoChou: articlequality: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842362 (https://phabricator.wikimedia.org/T318932) [09:47:58] onnit :) [09:48:23] (03CR) 10Klausman: [C: 03+1] extractor_utils: handle more exception cases [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842361 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [09:50:22] (03CR) 10AikoChou: [C: 03+1] extractor_utils: handle more exception cases [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842361 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [09:52:43] thanksss [09:55:29] latency in ml-serve-codfw seems better [09:55:39] p95 is around 2s [09:56:06] (03CR) 10Elukey: [C: 03+2] extractor_utils: handle more exception cases [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842361 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [10:05:47] folks I noticed one thing [10:05:58] in ORES, euwiki is reported [10:06:21] among errors, but I see that we don't have it on lift wing (at least for goodfaith) [10:06:51] ah no wait I see euwiki in scores processed for ORES [10:07:06] do we support the same number of wikis between ORES and Lift Wing? [10:08:00] ah right we don't have euwiki for goodfaith [10:08:06] (03Merged) 10jenkins-bot: extractor_utils: handle more exception cases [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842361 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [10:08:07] but for article quality etc.. [10:12:50] testing more wikis with good faith [10:13:09] (basically all the ones that we support) [10:22:49] (03CR) 10Ladsgroup: [C: 03+2] Various minor code cleanups [extensions/ORES] - 10https://gerrit.wikimedia.org/r/836827 (owner: 10Thiemo Kreuz (WMDE)) [10:25:44] (03Merged) 10jenkins-bot: Various minor code cleanups [extensions/ORES] - 10https://gerrit.wikimedia.org/r/836827 (owner: 10Thiemo Kreuz (WMDE)) [10:26:31] It would be really nice to have a kinda matrix of which wikis are supported by which models. And possibly a bit of a blurb on whether future support is planned (or cancellation of support) [10:27:27] (03CR) 10Elukey: [C: 03+1] articlequality: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842362 (https://phabricator.wikimedia.org/T318932) (owner: 10AikoChou) [10:29:31] klausman: in theory that should be taken care by the model cards IIUC [10:32:26] Good point [10:33:37] but we'll need to be good in keeping it in sync with production deployments etc.. [10:34:06] Well, it should definitely be the source of "intent" [10:34:27] (03PS1) 10AikoChou: events.py: fix revision score generation code [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842414 [10:35:24] (03PS2) 10AikoChou: events.py: fix revision score generation code [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842414 [10:37:29] 10Machine-Learning-Team, 10Data-Engineering, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10elukey) >>! In T319214#8307578, @Ottomata wrote: > The tricky thing about async calls in streams, is that the ordering of the events might get all messed up, as th... [10:37:37] elukey: ^^^ I found a small bug when I was testing the output of articletopic [10:40:30] aiko: ah nice thanks! Going to review it after lunch! [10:44:30] * elukey lunch [10:44:38] elukey: np! [10:46:38] (03PS2) 10AikoChou: topic: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842360 (https://phabricator.wikimedia.org/T318932) [10:47:30] (03PS2) 10AikoChou: articlequality: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842362 (https://phabricator.wikimedia.org/T318932) [10:48:44] (03PS2) 10AikoChou: draftquality: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842358 (https://phabricator.wikimedia.org/T318932) [11:00:16] <- lunch [11:58:22] elukey: ah it won't be correct for editquality because the pred is boolean type.. hmm [12:01:06] elukey: I need to think about how to solve it [12:49:19] aiko: ack! [13:28:32] elukey: i just invited you to our event platform sync meeting on thursday the 20th. we wanna sync up with you about ml and streaming and bethos and python, etc. [13:28:52] feel free to schedule another time if you can' tmake it, and/or invite anyone else you think would be useful [13:30:24] ottomata: o/ sure I'll join! [13:30:45] Filippo would be probably interested as well [13:45:15] (03PS1) 10Elukey: extractor_utils: add missing UnexpectedContentType import [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842450 (https://phabricator.wikimedia.org/T320374) [13:45:25] * elukey hides in shame [13:45:57] (03CR) 10Elukey: "/me hides in shame" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842450 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:55:32] (03CR) 10Elukey: [C: 03+2] "Merging since it is a straightforward patch, it should have been part of the previous one sorry :(" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842450 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:59:51] It's alright! :D [14:02:44] (03Merged) 10jenkins-bot: extractor_utils: add missing UnexpectedContentType import [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842450 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [14:40:05] I am testing drafttopic, draftquality and articletopic with benthos [14:40:14] so far nothing weird popped up [14:50:05] of course problem found :D [14:50:12] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/842465/ [14:50:29] we don't have EventGate settings for articletopic [14:50:30] so it fails [14:59:40] aaah, egress :) [15:39:10] ok so now everything works nicely [15:50:59] nice \o/ [15:51:51] aiko: when you have a moment can you tell me what you think about https://phabricator.wikimedia.org/T320374#8314266 ? [15:51:58] (not urgent even tomorrow) [15:54:07] ok I'll take a look! [15:59:49] thanks! It is basically the only weird thing that I spotted, articlequality + wikidata [16:12:09] going afk for the evening folks! [16:12:14] have a nice rest of the day :) [18:02:11] (03PS3) 10AikoChou: articlequality: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842362 (https://phabricator.wikimedia.org/T318932) [18:02:39] (03CR) 10AikoChou: [C: 03+2] articlequality: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842362 (https://phabricator.wikimedia.org/T318932) (owner: 10AikoChou) [18:09:32] (03Merged) 10jenkins-bot: articlequality: align ORES prediction output with Lift Wing's one [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/842362 (https://phabricator.wikimedia.org/T318932) (owner: 10AikoChou)