[02:43:09] 07artificial-intelligence, 10Reconciliation, 10Technical-Tool-Request: Alternative, affordable, lower-barrier approach(es) to reconciliation - https://phabricator.wikimedia.org/T362149#9833693 (10Thadguidry) One of our plans with [[ https://db2rest.org | DB2Rest ]] is to provide a simple instant Recon API fo... [05:58:48] Good morning folks o/ [07:10:50] 06Machine-Learning-Team: Allow calling revertrisk language agnostic and revert risk multilingual APIs in a pre-save context - https://phabricator.wikimedia.org/T356102#9833865 (10kostajh) >>! In T356102#9794121, @achou wrote: > Thanks for sharing the use case! >> Potentially called on all edit attempts by not-ye... [07:13:11] 06Machine-Learning-Team, 06Moderator-Tools-Team, 06Research, 10Temporary accounts, 06Trust and Safety Product Team: RevertRisk model readiness for temporary accounts - https://phabricator.wikimedia.org/T352839#9833868 (10kostajh) Note that per {T359405}, all calls to the revert risk models would be linke... [07:40:30] 06Machine-Learning-Team: Have problem with migrating to LiftWing from ores - https://phabricator.wikimedia.org/T364089#9833961 (10kostajh) 05Open→03Resolved Please re-open if there is still an issue. [09:12:49] 06Machine-Learning-Team: Tweak partman recipe for ML k8s workers - https://phabricator.wikimedia.org/T365971 (10klausman) 03NEW [09:13:21] 06Machine-Learning-Team: Tweak partman recipe for ML k8s workers - https://phabricator.wikimedia.org/T365971#9834217 (10klausman) [09:13:24] 06Machine-Learning-Team, 05Goal: 2024 Q4 Goal: Operational Excellence - Improve base monitoring, alerting and logging of Lift Wing services - https://phabricator.wikimedia.org/T362674#9834216 (10klausman) [09:14:11] Morning! [09:20:34] o/ Tobias! [09:20:43] Going for an early lunch! [09:25:40] good morning :) [09:33:50] 你好 :) [09:48:11] * klausman also lunch [10:25:48] hey aiko! [12:36:39] 06Machine-Learning-Team: Investigate why article-descriptions LiftWing API returns 404 when encoded colon is used in request URL - https://phabricator.wikimedia.org/T365439#9834783 (10hnowlan) The normalisation change has unfortunately not fixed this issue - docs indicate that it should have but I suspect this i... [12:49:09] folks in alerts.wikimedia.org I noticed that there were probe downs for the staging control plane nodes (ml-staging-ctrl2*), puppet was disabled for maintenance and there were some ferm/firewall changes [12:49:41] remember to check alerts every now and then :) [12:50:39] hey Luca o/ thanks for mentioning [12:51:15] I think we should set up some irc notification to be sure [12:52:01] this one is part of the main SRE alerts [12:52:17] related to failed systemd units, probes not answering, etc.. [12:52:24] maybe the probe one could be re-routed [12:52:36] but in general, keep an eye for ml* on alerts.w.o [12:55:50] ack [13:01:03] (03PS1) 10Kevin Bazira: test: add locust load test [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1035868 (https://phabricator.wikimedia.org/T365554) [13:02:46] (03CR) 10CI reject: [V:04-1] test: add locust load test [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1035868 (https://phabricator.wikimedia.org/T365554) (owner: 10Kevin Bazira) [13:16:32] (03PS2) 10Kevin Bazira: test: add locust load test [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1035868 (https://phabricator.wikimedia.org/T365554) [13:17:45] (03CR) 10CI reject: [V:04-1] test: add locust load test [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1035868 (https://phabricator.wikimedia.org/T365554) (owner: 10Kevin Bazira) [13:25:11] (03PS3) 10Kevin Bazira: test: add locust load test [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1035868 (https://phabricator.wikimedia.org/T365554) [13:25:27] finally the moment came for me to catch up with pydantic v2 breaking changes [13:26:00] as I'm trying to add validation for the payload in liftwing-python package [13:26:23] (03CR) 10CI reject: [V:04-1] test: add locust load test [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1035868 (https://phabricator.wikimedia.org/T365554) (owner: 10Kevin Bazira) [13:30:25] (03PS4) 10Kevin Bazira: test: add locust load test [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1035868 (https://phabricator.wikimedia.org/T365554) [13:34:42] (03CR) 10Kevin Bazira: "rec-api-ng load tests have borrowed functionality we use in the locust load tests for LW isvcs." [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1035868 (https://phabricator.wikimedia.org/T365554) (owner: 10Kevin Bazira) [14:30:27] (03PS3) 10Rockingpenny4: Adds article topic model to ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) [14:30:28] * isaranto afk - be back in an hour [14:32:29] (03CR) 10CI reject: [V:04-1] Adds article topic model to ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) (owner: 10Rockingpenny4) [14:33:50] 06Machine-Learning-Team, 06Research: Add Article Quality Model to LiftWing - https://phabricator.wikimedia.org/T360455#9835164 (10Isaac) Just adding another note of where these quality scores could be useful (filtering machine translation candidates): T293648#9816202 [14:43:05] (03PS4) 10Rockingpenny4: Adds article topic model to ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) [14:44:09] (03CR) 10CI reject: [V:04-1] Adds article topic model to ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) (owner: 10Rockingpenny4) [14:45:33] (03PS5) 10Rockingpenny4: Adds article topic model to ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) [14:47:24] (03CR) 10CI reject: [V:04-1] Adds article topic model to ORES [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) (owner: 10Rockingpenny4) [15:37:30] * isaranto back! [16:27:15] 06Machine-Learning-Team: Add pydantic validation to revertrisk model in liftwing-python package - https://phabricator.wikimedia.org/T366015 (10isarantopoulos) 03NEW [16:33:26] (03CR) 10Rockingpenny4: Adds article topic model to ORES (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) (owner: 10Rockingpenny4) [16:33:39] 06Machine-Learning-Team: Add pydantic validation to revertrisk model in liftwing-python package - https://phabricator.wikimedia.org/T366015#9835693 (10isarantopoulos) This is the relevant Pull Request : https://github.com/wikimedia/liftwing-python/pull/5 [16:34:33] aiko: o/ could you review this some time during the week? https://github.com/wikimedia/liftwing-python/pull/5 [16:35:01] iirc you've seen pydantic v2 a bit [16:36:09] perhaps this PR could be broken into 2-3 separate ones, if you wish I can do it so that we can focus on different parts each time [17:35:55] ok, for huggingface image I figured out how to set the command directly in the deployment charts [17:38:00] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1036297 [17:38:45] I tested it successfully with bert. with mistral the server seems to start properly but I see 1/3 containers and when I try to make a request it returns empty. I'm a bit puzzled but I'll dig more [17:44:36] ok, mistral started properly after a while , getting some errors which I think are specific to the model [17:48:46] 06Machine-Learning-Team, 13Patch-For-Review: Upgrade Huggingface image to kserve 0.13-rc0 (torch 2.3.0 ROCm 6.0) - https://phabricator.wikimedia.org/T365246#9835940 (10isarantopoulos) After defining `--backed=hugginface` in the entrypoint command the server starts properly but I'm getting an error when I make... [17:49:04] going afk folks, cu next week <3 [20:34:00] (03CR) 10Sohom Datta: "@rockingpenny4@gmail.com Doesn't this function run for all scores being fetched? Do we want to" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) (owner: 10Rockingpenny4) [20:34:45] (03CR) 10Sohom Datta: "(Ignore this comment!)" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) (owner: 10Rockingpenny4) [20:36:35] (03CR) 10Sohom Datta: "For the sake of documenting the technical decisions here, we are trying to only store the predicted topic and not all the probabilities si" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1035044 (https://phabricator.wikimedia.org/T218132) (owner: 10Rockingpenny4)