[03:51:17] 10Lift-Wing, 06Machine-Learning-Team, 13Patch-For-Review: Request to host the Reference Need Model on LiftWing - https://phabricator.wikimedia.org/T371902#10115785 (10Aitolkyn) [03:57:06] 10Lift-Wing, 06Machine-Learning-Team, 13Patch-For-Review: Request to host the Reference Need Model on LiftWing - https://phabricator.wikimedia.org/T371902#10115797 (10Aitolkyn) Hello @isarantopoulos! We downgraded to match the version in the knowledge-integrity repo. [05:56:21] (03PS1) 10Santhosh: Initialize the cache on application startup [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070481 [06:02:27] (03PS2) 10Santhosh: Initialize the cache on application startup [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070481 [07:15:49] good morning! [07:16:03] Hello! [07:59:28] (03CR) 10Ilias Sarantopoulos: [C:03+1] "Works like a charm, nice work!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1070060 (https://phabricator.wikimedia.org/T371902) (owner: 10AikoChou) [08:01:03] aiko: the patch works great! I'll follow up with research in the task about the dependencies I mentioned [08:01:45] I think that we need to bump them somehow cause the versions are getting too old. At least for new stuff it is good to use later versions [08:02:13] e.g. debian trixie come with python 3.12 already so that can be an issue further down the road [08:33:04] I filed a change to temporarily remove the articlequality deployments from prod until all details for schemas are finalized https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1070535 [08:33:19] just to be sure nobody starts using it [08:33:34] once we have all the details everything will be ready to go! [08:34:35] yeah I also think it'll be good to use a newer version [08:35:45] although it might be painful to bump them [08:36:34] +1ed for the articlequality change [08:46:55] ack [08:47:10] I updated the patch cause it would also remove the staging deployment [08:55:03] oops I missed it too haha [08:56:16] it was a test :P [09:01:11] (03CR) 10Kevin Bazira: [C:03+1] "Thank you for working on this, Aiko. I've tested this commit on linux and it run successfully:" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1070060 (https://phabricator.wikimedia.org/T371902) (owner: 10AikoChou) [09:12:48] I removed the articlequality deployments for now! [09:16:27] (from prod) [09:28:06] 06Machine-Learning-Team: Allow calling revertrisk language agnostic and revert risk multilingual APIs in a pre-save context - https://phabricator.wikimedia.org/T356102#10116437 (10achou) > A question before resolving this task--do you have any dashboards set up in Grafana for monitoring the latency of the pre-sa... [10:03:08] (03CR) 10AikoChou: [WIP] Add AbuseFilter variable for revertrisk score (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1051837 (https://phabricator.wikimedia.org/T364705) (owner: 10Kosta Harlan) [10:05:34] * aiko lunch ^^ [10:08:11] 06Machine-Learning-Team: Allow calling revertrisk language agnostic and revert risk multilingual APIs in a pre-save context - https://phabricator.wikimedia.org/T356102#10116619 (10diego) @achou, just for my curiosity, is the "predict time" the total end-to-end period or total = preprocess + predict? [10:14:12] 06Machine-Learning-Team: Allow calling revertrisk language agnostic and revert risk multilingual APIs in a pre-save context - https://phabricator.wikimedia.org/T356102#10116655 (10achou) Hi @diego! Total = preprocess + predict [10:14:44] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 06Growth-Team, 10MediaWiki-Recent-changes, and 2 others: Enable Revert Risk RecentChanges filter on id.wiki - https://phabricator.wikimedia.org/T365701#10116662 (10Samwalton9-WMF) >>! In T365701#10113596, @Scardenasmolinar wrote: > @Samwalton9-WMF shoul... [10:31:32] * isaranto afk lunch [10:39:39] * klausman lunch as well [11:46:03] (03PS1) 10Kevin Bazira: locust: add Makefile to run locust load tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1070575 (https://phabricator.wikimedia.org/T369728) [11:50:26] (03CR) 10Kevin Bazira: "I've tested this on stat1008 by running:" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1070575 (https://phabricator.wikimedia.org/T369728) (owner: 10Kevin Bazira) [13:14:15] FIRING: ErrorBudgetBurn: - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [13:40:33] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install ml-serve1009-1011 (3x), ml-lab1001-1002 (2x), dse-k8s-worker1009 (1x) - https://phabricator.wikimedia.org/T372432#10117416 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab... [14:36:25] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install ml-serve1009-1011 (3x), ml-lab1001-1002 (2x), dse-k8s-worker1009 (1x) - https://phabricator.wikimedia.org/T372432#10117659 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001... [15:07:08] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install ml-serve1009-1011 (3x), ml-lab1001-1002 (2x), dse-k8s-worker1009 (1x) - https://phabricator.wikimedia.org/T372432#10117845 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab... [15:51:05] 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399#10118097 (10Ottomata) [15:52:28] 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399#10118094 (10Ottomata) [16:19:42] (03CR) 10Ilias Sarantopoulos: "Nice work!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1070575 (https://phabricator.wikimedia.org/T369728) (owner: 10Kevin Bazira) [16:45:04] 10Lift-Wing, 06Machine-Learning-Team: Log and export preprocess size in inference services as a prometheus metric - https://phabricator.wikimedia.org/T374034 (10isarantopoulos) 03NEW [17:14:15] FIRING: ErrorBudgetBurn: - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [17:28:01] * isaranto afk! [19:07:55] 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install ml-serve1009-1011 (3x), ml-lab1001-1002 (2x), dse-k8s-worker1009 (1x) - https://phabricator.wikimedia.org/T372432#10119256 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001... [21:14:15] FIRING: ErrorBudgetBurn: - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn