[06:11:36] (03PS1) 10Kevin Bazira: article-country: filter out empty country entries in wikilink predicitons [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1127793 (https://phabricator.wikimedia.org/T385970) [07:57:17] hello! [08:24:44] FIRING: LiftWingServiceErrorRate: ... [08:24:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=ptwiki-damaging-predictor-default.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [08:33:25] looking --^ [08:39:49] Good morning [09:18:44] RESOLVED: LiftWingServiceErrorRate: ... [09:18:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=revscoring-editquality-damaging&var-backend=ptwiki-damaging-predictor-default.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [09:38:44] it was resolved by itself. I saw some (a lot ) cpu throttling so it was the standard issue with revscoring preprocess. https://grafana.wikimedia.org/goto/6Y0o8chHg?orgId=1 [09:38:44] if it occurs again we can enable multiprocessing with the following patch https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1127851 [09:38:53] I set it as WIP for now [09:44:41] Morning! [09:46:57] morning Tobias! [09:53:11] o/ whenever anyone gets a minute, please review: https://gerrit.wikimedia.org/r/1127793 [09:53:11] thanks! [09:53:44] (03CR) 10Klausman: [C:03+1] article-country: filter out empty country entries in wikilink predicitons [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1127793 (https://phabricator.wikimedia.org/T385970) (owner: 10Kevin Bazira) [09:54:00] Wasn't on the reviewers list, but LGTM :) [09:56:12] (03CR) 10Kevin Bazira: [C:03+2] article-country: filter out empty country entries in wikilink predicitons [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1127793 (https://phabricator.wikimedia.org/T385970) (owner: 10Kevin Bazira) [09:56:20] klausman: thanks :) [09:56:58] (03Merged) 10jenkins-bot: article-country: filter out empty country entries in wikilink predicitons [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1127793 (https://phabricator.wikimedia.org/T385970) (owner: 10Kevin Bazira) [10:33:33] 06Machine-Learning-Team, 10EditCheck, 10Editing-team (Tracking): Verify cost of gathering peacock training/evaluation data for top 20 languages - https://phabricator.wikimedia.org/T388215#10635677 (10matej_suchanek) In Czech Wikipedia, I think there is no commonly understood term for "peacock words." It simp... [10:50:09] * isaranto afk lunch [11:25:25] * klausman lunch as well [13:07:41] anyone with a moment, please review: https://gerrit.wikimedia.org/r/1127857 [13:07:41] thanks! [13:16:31] kevinbazira: +1! [13:27:32] aiko: 谢谢 :) [13:46:19] 10Lift-Wing, 06Machine-Learning-Team, 07OKR-Work, 13Patch-For-Review: Update the article-country isvc to use Wikilinks for predictions - https://phabricator.wikimedia.org/T385970#10636225 (10kevinbazira) >>! In T385970#10633019, @Isaac wrote: > One small thing that I've noticed -- empty string countries ca... [13:55:07] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck: Load test the peacock edit check service - https://phabricator.wikimedia.org/T388817#10636240 (10gkyziridis) **Edit-Check Service - Peacock model CPU version TESTS** {P74228} [13:59:48] (03PS1) 10Gkyziridis: inference-services: Edit-check service locust tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1127904 (https://phabricator.wikimedia.org/T388817) [14:03:14] Hey folks \o [14:03:49] What would be a nice result for locust tests for a service on average? Feel free to comment on https://phabricator.wikimedia.org/T388817 [14:20:30] I remember the requirement from product team is within 500ms [14:24:21] (03CR) 10AikoChou: [C:03+1] "Only the *_stats.csv needs to be committed. Other than that LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1127904 (https://phabricator.wikimedia.org/T388817) (owner: 10Gkyziridis) [14:29:45] (03PS2) 10Gkyziridis: inference-services: Edit-check service locust tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1127904 (https://phabricator.wikimedia.org/T388817) [14:31:53] (03CR) 10Gkyziridis: [C:03+2] "Merging" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1127904 (https://phabricator.wikimedia.org/T388817) (owner: 10Gkyziridis) [14:39:40] (03CR) 10Gkyziridis: [V:03+2 C:03+2] inference-services: Edit-check service locust tests. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1127904 (https://phabricator.wikimedia.org/T388817) (owner: 10Gkyziridis) [14:44:23] https://www.irccloud.com/pastebin/mk18Vtnd/ [14:44:50] We are on average 1000 ms [14:48:13] I hope that when we will develop it on GPUs will be faster [14:52:54] yeah we want under 500ms but don't worry about the latencies on the CPU [14:54:51] alrighty [15:36:07] 10Lift-Wing, 06Machine-Learning-Team, 07OKR-Work, 13Patch-For-Review: Update the article-country isvc to use Wikilinks for predictions - https://phabricator.wikimedia.org/T385970#10636566 (10Isaac) Hey -- as we discussed separately, let's update the code to still include the tfidf value for the empty count... [21:12:37] 06Machine-Learning-Team, 10EditCheck, 10VisualEditor, 10Editing-team (Kanban Board): Evaluate efficacy of Peacock Check model output (internal review) - https://phabricator.wikimedia.org/T384651#10637791 (10ppelberg) [21:17:11] 06Machine-Learning-Team, 10EditCheck, 10VisualEditor, 10Editing-team (Kanban Board): Evaluate efficacy of Peacock Check model output (internal review) - https://phabricator.wikimedia.org/T384651#10637815 (10ppelberg) [21:18:07] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck: Investigate options for providing beta cluster / patchdemo access to liftwing staging - https://phabricator.wikimedia.org/T388269#10637818 (10ppelberg) [21:29:32] 06Machine-Learning-Team, 10EditCheck: Evaluate the existing peacock detection model - https://phabricator.wikimedia.org/T386645#10637882 (10ppelberg) [21:32:31] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck: Create dummy peacock detection model server - https://phabricator.wikimedia.org/T386100#10637906 (10ppelberg) [21:37:17] 06Machine-Learning-Team, 10EditCheck, 10VisualEditor, 10Editing-team (Kanban Board): Evaluate efficacy of Peacock Check model output (internal review) - https://phabricator.wikimedia.org/T384651#10637937 (10ppelberg)