[07:20:37] Good morning! [07:51:06] morning Ilias! hope you feel better now [07:58:44] yep! a good night's sleep can do wonders! [08:05:59] (03PS3) 10AikoChou: Add Makefile for local-run [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/988666 (https://phabricator.wikimedia.org/T352689) [08:06:12] nice :) [08:32:16] (03PS4) 10AikoChou: Add Makefile for local-run [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/988666 (https://phabricator.wikimedia.org/T352689) [08:35:55] (03CR) 10AikoChou: "Also updated the README.md. Thanks for the review :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/988666 (https://phabricator.wikimedia.org/T352689) (owner: 10AikoChou) [08:36:50] (03CR) 10AikoChou: [C: 03+2] Add Makefile for local-run [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/988666 (https://phabricator.wikimedia.org/T352689) (owner: 10AikoChou) [09:11:52] kevinbazira: task T351939 is intended to just commit the load test results in the repo so that we have it as reference. We can keep the comparison in a separate task. I'm working on a POC of how we can do the comparison which we can then apply to all model servers [09:15:26] o/ there's a new release for knowledge integrity https://gitlab.wikimedia.org/repos/research/knowledge_integrity/-/releases/v0.6.0 [09:15:31] 10Machine-Learning-Team: Investigate way of comparing load test results - https://phabricator.wikimedia.org/T355394 (10isarantopoulos) [09:15:41] some module name changes, so our inference code will need adjustments [09:17:13] load testing is also needed to ensure everything is working properly [09:17:25] will open a task for that [09:18:28] 10Machine-Learning-Team: Investigate way of comparing load test results - https://phabricator.wikimedia.org/T355394 (10isarantopoulos) a:03isarantopoulos [09:18:38] 10Machine-Learning-Team: Investigate way of comparing load test results - https://phabricator.wikimedia.org/T355394 (10isarantopoulos) [09:19:34] 10Machine-Learning-Team, 10Patch-For-Review: Establish a standard load testing procedure - https://phabricator.wikimedia.org/T348850 (10isarantopoulos) p:05Triageā†’03Medium [09:19:59] 10Machine-Learning-Team: Investigate way of comparing load test results - https://phabricator.wikimedia.org/T355394 (10isarantopoulos) p:05Triageā†’03Medium [09:20:37] ack aiko: ! [09:21:12] I also keep checking catboost waiting for that new release.. [09:22:52] 10Machine-Learning-Team, 10Moderator-Tools-Team, 10Research, 10Temporary accounts, 10Trust and Safety Product Team: RevertRisk model readiness for temporary accounts - https://phabricator.wikimedia.org/T352839 (10kostajh) >>! In T352839#9471063, @diego wrote: > @MunizaA , until we don't have enough train... [09:29:32] isaranto: their last release was in September last year.. [09:30:58] yes. we are waiting for the new release that includes Luca's PR https://github.com/catboost/catboost/pull/2519 which has been merged [09:31:46] I said that I'll follow up and ask about it, so I'll do it today [09:32:28] thanks! [11:43:52] * klausman lunch and doc appt [11:48:46] (03PS3) 10Kosta Harlan: WIP revertrisk: Add thresholds for likely/very likely bad [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [12:16:54] (03PS4) 10Kosta Harlan: WIP revertrisk: Rename model [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [12:17:00] (03PS1) 10Kosta Harlan: tests: Use @covers at class level [extensions/ORES] - 10https://gerrit.wikimedia.org/r/991762 [12:17:06] (03PS1) 10Kosta Harlan: Helpers: Check for underscore and dashes when building query [extensions/ORES] - 10https://gerrit.wikimedia.org/r/991763 [12:19:09] (03CR) 10CI reject: [V: 04-1] WIP revertrisk: Rename model [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [12:19:16] (03CR) 10CI reject: [V: 04-1] Helpers: Check for underscore and dashes when building query [extensions/ORES] - 10https://gerrit.wikimedia.org/r/991763 (owner: 10Kosta Harlan) [12:21:08] I've made some nice progress with merging the results in order to start the comparison . I'll update my patch before EOD to share more [12:21:16] * isaranto going for lunch! [12:23:24] (03PS2) 10Kosta Harlan: Helpers: Check for underscore and dashes when building query [extensions/ORES] - 10https://gerrit.wikimedia.org/r/991763 [12:23:27] (03PS5) 10Kosta Harlan: WIP revertrisk: Rename model [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [12:31:54] (03PS6) 10Kosta Harlan: WIP revertrisk: Rename model to use single lowercase word [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [12:33:06] (03PS7) 10Kosta Harlan: WIP revertrisk: Rename model to use single lowercase word [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [12:35:12] (03CR) 10CI reject: [V: 04-1] WIP revertrisk: Rename model to use single lowercase word [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [12:35:18] (03PS8) 10Kosta Harlan: WIP revertrisk: Rename model to use single lowercase word [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [12:36:04] (03PS9) 10Kosta Harlan: WIP revertrisk: Rename model to use single lowercase word [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [13:11:50] (03PS10) 10Kosta Harlan: WIP revertrisk: Rename model to use single lowercase word [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [13:27:16] (03CR) 10AikoChou: [V: 03+2 C: 03+2] Add Makefile for local-run [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/988666 (https://phabricator.wikimedia.org/T352689) (owner: 10AikoChou) [13:34:24] (03PS11) 10Kosta Harlan: WIP revertrisk: Rename model to use single lowercase word [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [13:39:56] 10Machine-Learning-Team: Document load test results - https://phabricator.wikimedia.org/T351939 (10kevinbazira) In order to compare historical data from T351939#9469592, I updated [[ https://phabricator.wikimedia.org/P55025 | article_descriptions.py ]] with `lw_stats_analysis()` and changed `lw_stats_history()`... [13:51:53] isaranto: o/ [13:51:53] I've completed a prototype that adds an easily digestible LiftWing-specific comparison after a locust load test has concluded: [13:51:53] https://phabricator.wikimedia.org/T351939#9472642 [13:51:53] happy to discuss this with you on Tuesday when you're back :) [14:02:41] Ok I did sth similar with creating pandas df and joining the results. I'm looking if we can get the raw latencies (of each request) rather than the aggregated stats so we can run staitic tests on them to test if the changes we see are statistically significant [14:20:35] have a nice weekend Kevin :) [14:32:01] I used aggregated stats because they provided average response times and failure counts for each load test run. This made it easy to compare the results of previous load tests. [14:32:01] enjoy your weekend Ilias :) [14:34:20] Good morning all [14:34:23] Okay Iā€™m back [14:34:49] I slept for like 14 hours total yesterday [14:34:57] hey Chris, happy to see you're back! feeling better? [14:36:04] ohey Chris [14:45:03] (03PS12) 10Kosta Harlan: revertrisk: Rename model to use single lowercase word [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) [14:54:46] one can take a sneak peak in the way I used to merge old and new load test results in order to compare them https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/989732 [14:55:13] it is WIP. but we should coordinate a bit better as we're working on the same thing... [14:56:12] I propose we create a robust solution for 1 model server and then we execute this for all of them. [14:56:42] isaranto: I merged our two patches and kept the deployExtraClusterRoles gate, so we can make sure even if someone else has an experimental NS, they won't accidentally give us privileges on it. [14:57:42] awesome klausman: ! I'm reviewing now [14:58:06] I also added Alex as a reviewer and will ping him if he's the right one to review, or who else might be [15:03:23] thanks Tobias for this work. finally I understood how deployClusterRoles comes into play. [15:06:25] Yeah, that took me a long while to grok as well [15:07:08] It's confusing that it _looks_ like it's a thing that k8s or Helm ship with, when in reality it's just a WMF-invented mechanism using YAML vars [15:07:18] well, strictly speaking Go template vars, but [15:07:51] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "LGTM!" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/988048 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [15:31:02] (03CR) 10Klausman: [C: 03+1] Add Makefile for local-run [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/988666 (https://phabricator.wikimedia.org/T352689) (owner: 10AikoChou) [16:02:47] Going afk. Have a nice weekend folks o/ [16:05:00] \o [18:35:37] (03CR) 10DannyS712: [C: 03+2] tests: Use @covers at class level [extensions/ORES] - 10https://gerrit.wikimedia.org/r/991762 (owner: 10Kosta Harlan) [18:56:50] (03Merged) 10jenkins-bot: tests: Use @covers at class level [extensions/ORES] - 10https://gerrit.wikimedia.org/r/991762 (owner: 10Kosta Harlan) [19:03:17] 10Machine-Learning-Team, 10Moderator-Tools-Team, 10Research, 10Temporary accounts, 10Trust and Safety Product Team: RevertRisk model readiness for temporary accounts - https://phabricator.wikimedia.org/T352839 (10Tchanders) >>! In T352839#9471909, @kostajh wrote: >>>! In T352839#9471063, @diego wrote: >>... [22:29:25] (03CR) 10Jdlrobson: [C: 03+1] Don't use live configuration [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman) [22:52:13] (03CR) 10Jsn.sherman: [C: 03+2] "Thank you for following up on this after I let it rot; it's a little odd for me to +2 my own patch, but we've got multiple contributors an" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman) [23:27:30] (03Merged) 10jenkins-bot: Don't use live configuration [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman)