[06:50:59] <isaranto>	 Good morning o/
[08:39:27] <aiko>	 morning! :)
[08:41:48] <isaranto>	 Hey Aiko!
[08:41:57] <isaranto>	 Afk for 30'!
[09:30:49] <isaranto>	 kevinbazira: aiko: since I'm mostly blocked until we fix the permissions on ml-staging shall I be of any help in either load testing or batcher?
[09:31:09] <isaranto>	 kevinbazira: let's discuss how we are going to compare load test results before we jump into implementing
[09:31:26] <isaranto>	 do you have something specific in mind?
[09:31:27] <kevinbazira>	 isaranto: o/
[09:33:23] <kevinbazira>	 The files produced by a locust load test I did in T351939 are:
[09:33:23] <kevinbazira>	 `article_descriptions_exceptions.csv`, `article_descriptions_stats_history.csv`, `article_descriptions_failures.csv`, and `article_descriptions_stats.csv`
[09:33:23] <kevinbazira>	 I am studying them to understand which ones will have the data that we'll want to compare.
[09:58:03] <wikibugs>	 10Machine-Learning-Team: Improving error message for Revertrisk models - https://phabricator.wikimedia.org/T351278 (10achou) p:05Medium→03Triage
[09:58:19] <wikibugs>	 10Machine-Learning-Team: Improving error message for Revertrisk models - https://phabricator.wikimedia.org/T351278 (10achou) p:05Triage→03Medium
[09:58:46] <wikibugs>	 10Machine-Learning-Team: Improving error message for Revertrisk models - https://phabricator.wikimedia.org/T351278 (10achou) p:05Medium→03Triage
[10:03:30] <isaranto>	 ok, I can look into the comparison as well + figure out if there is any connectivity to prod cluster from CI (I doubt it though)
[10:14:37] <kevinbazira>	 sure sure. I'll add updates to the task so that the findings are consolidated in one place.
[10:29:29] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: Set SLO for the recommendation-api-ng service hosted on LiftWing - https://phabricator.wikimedia.org/T347262 (10klausman) SLO Dashboard now available here: https://grafana.wikimedia.org/d/slo-Lift_Wing_Recommendation_API_NG/lift-wing-recommendation-api-ng-slo-s?orgId=1
[10:29:42] <klausman>	 Morning :)
[10:29:59] <klausman>	 Recom-api-ng SLO dashboard no available here: https://grafana.wikimedia.org/d/slo-Lift_Wing_Recommendation_API_NG/lift-wing-recommendation-api-ng-slo-s?orgId=1
[10:32:10] <isaranto>	 Guten tag Tobias!
[10:32:21] <wikibugs>	 10Machine-Learning-Team, 10SRE, 10Patch-For-Review: Requesting write access to ml-staging-codfw for ML team - https://phabricator.wikimedia.org/T354516 (10klausman) p:05Triage→03High
[10:32:38] <klausman>	 \o Καλημέρα!
[10:42:21] <isaranto>	 :)
[10:46:47] <wikibugs>	 10Machine-Learning-Team: Document load test results - https://phabricator.wikimedia.org/T351939 (10kevinbazira) I have been studying the files produced in T351939#9466020 and it looks like `article_descriptions_stats.csv` contains the fields that we'll want to compare: |Type|Name...
[11:30:57] * isaranto lunch
[12:18:00] <klausman>	 ditto
[14:13:35] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: WIP: test ci connectivity [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/991601
[15:09:35] <isaranto>	 kevinbazira: the easiest way I've found for comparison so far is by running the tests and then using another script to read the 2 csv files (the new and the old)  and then compare results. I'm looking though if we can just run sth inside the locust tests after they finish which would be optimal
[15:12:22] <kevinbazira>	 isaranto: I found that locust has a `test_stop` event listener that we can use for this:
[15:12:22] <kevinbazira>	 https://docs.locust.io/en/stable/writing-a-locustfile.html#test-start-and-test-stop
[15:12:22] <kevinbazira>	 working on a quick prototype, I'll share it as soon as it's ready
[15:14:15] <isaranto>	 Thanks!I'm going to try that ,it wasn't clear for me if it refers to a single test or the whole test suite but trying out will tell us
[15:45:59] <wikibugs>	 10Machine-Learning-Team: Document load test results - https://phabricator.wikimedia.org/T351939 (10kevinbazira) locust has a [[ https://docs.locust.io/en/stable/writing-a-locustfile.html#test-start-and-test-stop | test_stop ]] event listener that can be used to read `article_descriptions_stats.csv` after it has...
[15:59:09] <kevinbazira>	 isaranto: sure sure. since locust generates a new csv file on every run, I have used the `test_stop` event listener to extract aggregated data from that csv file and save it into `lw_history_stats.csv`. this new file maintains a record of historical load test data that can be used to perform a comparative analysis.
[15:59:09] <kevinbazira>	 I have shared the details here: https://phabricator.wikimedia.org/T351939#9469592
[20:49:44] <wikibugs>	 10Machine-Learning-Team, 10Moderator-Tools-Team, 10Research, 10Temporary accounts, 10Trust and Safety Product Team: RevertRisk model readiness for temporary accounts - https://phabricator.wikimedia.org/T352839 (10MunizaA) >>! In T352839#9467079, @diego wrote: >> Ideally, by the time we are deploying to p...
[22:18:49] <wikibugs>	 10Machine-Learning-Team, 10Moderator-Tools-Team, 10Research, 10Temporary accounts, 10Trust and Safety Product Team: RevertRisk model readiness for temporary accounts - https://phabricator.wikimedia.org/T352839 (10diego) @MunizaA , until we don't have enough training data we should treat temporary account...