[02:16:41] 07artificial-intelligence, 10MediaWiki-extension-requests: Develop an extension to implement LLM summaries of changes - https://phabricator.wikimedia.org/T420303 (10Awesome_Aasim) 03NEW [05:13:19] (03PS1) 10Kevin Bazira: policy-violation: add configurable gpu_memory_utilization flag for gpt-oss-safeguard-20b model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1253762 (https://phabricator.wikimedia.org/T418350) [08:48:35] (03CR) 10Bartosz Wójtowicz: [C:03+1] "LGTM, after deployment let's test it with big input lengths to ensure everything works smooth even!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1253762 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [08:50:33] (03CR) 10Kevin Bazira: [C:03+2] policy-violation: add configurable gpu_memory_utilization flag for gpt-oss-safeguard-20b model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1253762 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [08:52:13] (03Merged) 10jenkins-bot: policy-violation: add configurable gpu_memory_utilization flag for gpt-oss-safeguard-20b model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1253762 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [09:06:42] 06Machine-Learning-Team, 06Product Safety and Integrity: Deploy CoPE-A on LiftWing - https://phabricator.wikimedia.org/T418832#11717422 (10BWojtowicz-WMF) After lowering the maximum input token length to 4096, we seem to be able to process all incoming requests. I will figure out optimizations we could make to... [09:30:17] (03PS1) 10Bartosz Wójtowicz: policy-violation: Extend CoPE server to return confidence scores. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254119 (https://phabricator.wikimedia.org/T418832) [10:13:34] 10Lift-Wing, 06Machine-Learning-Team, 06Editing-team (Tracking), 07ml-model-requests, 07OKR-Work (WE1 FY2025-26): Increase batch size in edit-check service - https://phabricator.wikimedia.org/T419527#11717635 (10gkyziridis) 05Open→03Resolved [10:23:25] 06Machine-Learning-Team: Improve logging on Liftwing - https://phabricator.wikimedia.org/T420327 (10gkyziridis) 03NEW [10:43:53] (03PS1) 10Kevin Bazira: policy-violation: add configurable max_model_len flag for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254147 (https://phabricator.wikimedia.org/T418350) [10:46:24] (03CR) 10Majavah: [C:03+2] build: Updating composer dependencies [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1246497 (owner: 10Libraryupgrader) [10:47:00] (03PS2) 10Kevin Bazira: policy-violation: add configurable max_model_len flag for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254147 (https://phabricator.wikimedia.org/T418350) [10:53:37] (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you!!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254147 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [10:54:41] (03CR) 10Kevin Bazira: [C:03+2] policy-violation: add configurable max_model_len flag for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254147 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [10:55:17] (03Merged) 10jenkins-bot: policy-violation: add configurable max_model_len flag for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254147 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [11:55:57] (03PS1) 10Kevin Bazira: policy-violation: add configurable block_size flag for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254171 (https://phabricator.wikimedia.org/T418350) [11:59:55] (03CR) 10Bartosz Wójtowicz: [C:03+1] policy-violation: add configurable block_size flag for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254171 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [12:26:34] 06Machine-Learning-Team: Edit Suggestions - Edit suggestion generation with pre-defined edit types - https://phabricator.wikimedia.org/T418102#11718199 (10achou) a:03achou [12:49:16] (03CR) 10Kevin Bazira: [C:03+2] policy-violation: add configurable block_size flag for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254171 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [12:49:59] (03Merged) 10jenkins-bot: policy-violation: add configurable block_size flag for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254171 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [14:00:13] 06Machine-Learning-Team, 10ORES, 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ORES tables from wikis without ORES - https://phabricator.wikimedia.org/T420093#11718703 (10Dreamy_Jazz) [15:30:12] (03PS1) 10Kevin Bazira: policy-violation: add AITER support to gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254230 (https://phabricator.wikimedia.org/T418350) [15:36:58] (03CR) 10Bartosz Wójtowicz: policy-violation: add AITER support to gpt model-server (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254230 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [15:41:17] (03CR) 10Kevin Bazira: policy-violation: add AITER support to gpt model-server (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254230 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [15:50:42] (03PS2) 10Kevin Bazira: policy-violation: add AITER support to gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254230 (https://phabricator.wikimedia.org/T418350) [15:52:34] (03CR) 10Kevin Bazira: policy-violation: add AITER support to gpt model-server (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254230 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [16:02:36] (03CR) 10Bartosz Wójtowicz: [C:03+1] policy-violation: add AITER support to gpt model-server (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254230 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [16:04:25] (03CR) 10Kevin Bazira: [C:03+2] policy-violation: add AITER support to gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254230 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [16:05:02] (03Merged) 10jenkins-bot: policy-violation: add AITER support to gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254230 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [16:41:49] 06Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team (Kanban), and 2 others: Enable revert risk filters for first batch of wikis: < 1000 monthly edits - https://phabricator.wikimedia.org/T411485#11719883 (10Kgraessle) 05In progress→03Stalled Moving th... [17:07:44] FIRING: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=edit-check&var-backend=edit-check-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [17:11:39] (03PS1) 10Kevin Bazira: policy-violation: add compilation_config with AITER optimizations for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254257 (https://phabricator.wikimedia.org/T418350) [17:12:44] RESOLVED: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=edit-check&var-backend=edit-check-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [17:24:27] klausman, dpogorzelski - o/ ml-serve2001 seems down, please have a look when you have a min [17:26:49] 07artificial-intelligence, 10Citoid: Citoid block needs information (supposedly Anubis, but single case to fix as blueprint) - https://phabricator.wikimedia.org/T420397 (10Elya) 03NEW [17:30:18] (03CR) 10Bartosz Wójtowicz: [C:03+1] policy-violation: add compilation_config with AITER optimizations for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254257 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [17:44:34] (03CR) 10Kevin Bazira: [C:03+2] policy-violation: add compilation_config with AITER optimizations for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254257 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [17:45:07] (03Merged) 10jenkins-bot: policy-violation: add compilation_config with AITER optimizations for gpt model-server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1254257 (https://phabricator.wikimedia.org/T418350) (owner: 10Kevin Bazira) [18:03:44] FIRING: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=edit-check&var-backend=edit-check-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [18:08:44] RESOLVED: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=edit-check&var-backend=edit-check-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [18:40:44] FIRING: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=edit-check&var-backend=edit-check-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [18:45:44] RESOLVED: LiftWingServiceErrorRate: LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=eqiad%20prometheus/k8s-mlserve&var-namespace=edit-check&var-backend=edit-check-predictor.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [18:53:07] 06Machine-Learning-Team, 06Research: AI/ML Model Request: Image auto-crop / focus point detection - https://phabricator.wikimedia.org/T419287#11720579 (10HNordeenWMF) [23:17:10] 06Machine-Learning-Team, 10ORES, 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ORES tables from wikis without ORES - https://phabricator.wikimedia.org/T420093#11721270 (10Ladsgroup) p:05Triage→03Medium