[00:19:01] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Editing Suggestions - api - https://phabricator.wikimedia.org/T427794#11983585 (10ppelberg)
[06:30:19] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26): Deploy CoPE-B-A4B on LiftWing - https://phabricator.wikimedia.org/T427497#11983927 (10kevinbazira) The [zentropi-ai/cope-b-a4b](https://huggingface.co/zentropi-ai/cope-b-a4b#getting-started) docs show we have 3 hosting options: zentropi API, vLLM, HF transfo...
[06:55:18] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Editing Suggestions - api - https://phabricator.wikimedia.org/T427794#11983966 (10OKarakaya-WMF)
[07:04:03] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26): Deploy CoPE-B-A4B on LiftWing - https://phabricator.wikimedia.org/T427497#11983972 (10kevinbazira) We have updated the [liftwing_client](https://gitlab.wikimedia.org/repos/product-safety-and-integrity/llm-content-filtering-and-tagging/-/merge_requests/17) to...
[07:05:15] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26): Deploy CoPE-B-A4B on LiftWing - https://phabricator.wikimedia.org/T427497#11983974 (10kostajh) >>! In T427497#11983972, @kevinbazira wrote: > We have updated the [liftwing_client](https://gitlab.wikimedia.org/repos/product-safety-and-integrity/llm-content-fi...
[07:22:55] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 07OKR-Work: Integrate Article Topic model with the new caching service - https://phabricator.wikimedia.org/T418493#11984032 (10BWojtowicz-WMF) I did some more digging on the 500s.  Big chunk of those errors stems from the fact that I was using ~year old `revision_id`, wh...
[07:47:50] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 07OKR-Work: Improve and optimize Article Topics model to better handle queries with `revision_id` - https://phabricator.wikimedia.org/T428127 (10BWojtowicz-WMF) 03NEW
[07:48:00] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 07OKR-Work: Integrate Article Topic model with the new caching service - https://phabricator.wikimedia.org/T418493#11984120 (10BWojtowicz-WMF) We've turned on the integration on production as well 🎉  Example of how to query Hoards is below, it is only reachable internall...
[07:50:47] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 07OKR-Work: Integrate Article Topic model with the new caching service - https://phabricator.wikimedia.org/T418493#11984124 (10BWojtowicz-WMF) 05Open→03Resolved
[08:28:24] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Eval pipeline for enwiki - https://phabricator.wikimedia.org/T425957#11984171 (10achou) @OKarakaya-WMF One small suggestion: would it make sense to put project updates/findings directly in the Phab task instead of only linking to the doc?  I find it m...
[08:49:03] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Eval pipeline for enwiki - https://phabricator.wikimedia.org/T425957#11984217 (10OKarakaya-WMF) sure, I can add the main findings to the tasks as well. In general I find it easier to have all outputs in one place rather than searching in multiple phab...
[10:00:58] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): ml-lab: Update pytorch-rocm to a current ROCm-compatible release - https://phabricator.wikimedia.org/T428023#11984502 (10isarantopoulos)
[10:01:44] <wikibugs>	 (03CR) 10Hashar: feat(liftwing-openapi-server): Serve OpenAPI specs via Apache httpd (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1297072 (https://phabricator.wikimedia.org/T427902) (owner: 10Gkyziridis)
[11:18:27] <wikibugs>	 (03PS11) 10Gkyziridis: feat(liftwing-openapi-server): Serve OpenAPI specs via Apache httpd [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1297072 (https://phabricator.wikimedia.org/T427902)
[11:44:46] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 05Goal, 07OKR-Work: Q2 FY2025-26 Goal: Generate a list of edit suggestions using machine learning - https://phabricator.wikimedia.org/T409863#11984770 (10achou)
[12:25:12] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 10Semantic Search, 06Traffic, 07Essential-Work: Transparent DNS Routing for LiftWing Services (eqiad vs Multi-DC) - https://phabricator.wikimedia.org/T422253#11984950 (10isarantopoulos)
[12:40:32] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - api - https://phabricator.wikimedia.org/T427794#11985034 (10OKarakaya-WMF) deployment fails with the following message. ` kubectl describe revision editing-suggestions-predictor-00001      Last Failure Info:       Exit Code:  1       Message:    2026-...
[12:41:01] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 10Semantic Search, 06Traffic, 07Essential-Work: Transparent DNS Routing for LiftWing Services (eqiad vs Multi-DC) - https://phabricator.wikimedia.org/T422253#11985035 (10DPogorzelski-WMF) i think the original question was whether we had a way to use on...
[12:56:16] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06Infrastructure-Foundations: Move all Machine Learning Docker images under the /ml prefix in the Docker Registry - https://phabricator.wikimedia.org/T428022#11985067 (10achou) @elukey Thanks for putting this together! The list looks good to me.  - amd-pytorch21 / 22 - m...
[12:57:35] <wikibugs>	 (03PS1) 10Ozge: Fix editing-suggestions KServe model download [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1297687 (https://phabricator.wikimedia.org/T427794)
[13:29:36] <wikibugs>	 (03Abandoned) 10Ozge: Fix editing-suggestions KServe model download [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1297687 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge)
[14:43:19] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - api - https://phabricator.wikimedia.org/T427794#11985799 (10OKarakaya-WMF) We can easily handle 250 req/sec (probably more). As I understand from addalink dashboards, we currently get ~7 req/sec from editing suggestions.    `  (venv) ozge@stat1010:~/r...
[14:55:29] <wikibugs>	 (03PS1) 10Ozge: Adds editing-suggestions locust test [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1297725 (https://phabricator.wikimedia.org/T427794)
[14:58:14] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: [draft] Access control for LiftWing LLM services exposed to external clients through REST Gateway - https://phabricator.wikimedia.org/T426749#11985884 (10Clement_Goubert)
[15:13:08] <wikibugs>	 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team, 07Essential-Work, 07Wikimedia-production-error: MentionRegexException in link-recommendation service - https://phabricator.wikimedia.org/T420255#11985955 (10OKarakaya-WMF) a:03OKarakaya-WMF
[15:44:03] <wikibugs>	 06Machine-Learning-Team, 10Add-Link-Structured-Task, 06Growth-Team, 07Essential-Work, 07Wikimedia-production-error: MentionRegexException in link-recommendation service - https://phabricator.wikimedia.org/T420255#11986025 (10OKarakaya-WMF) I think this happens when links have chars that we can not decode...