[05:16:13] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 07Essential-Work: Rebuild vLLM base images to use deb.debian.org in apt sources - https://phabricator.wikimedia.org/T429667#12063649 (10kevinbazira)
[05:16:15] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12063648 (10kevinbazira)
[05:16:55] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 07Essential-Work: Rebuild vLLM base images to use deb.debian.org in apt sources - https://phabricator.wikimedia.org/T429667#12063651 (10kevinbazira) 05Open→03Resolved
[06:35:17] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Update static descriptions. - https://phabricator.wikimedia.org/T430030#12063728 (10OKarakaya-WMF) Hey @ppelberg, just checking in—are you still planning to update the static descriptions? If not, would it make sense to close this ticket?
[07:45:24] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12063921 (10achou) A few GPU facts for the decision (gathered from deployment-charts, puppet, doc, gerrit — please correct me if anything's off):  **Current s...
[08:58:49] <jinxer-wm>	 FIRING: KubernetesDeploymentUnavailableReplicas: ...
[08:58:55] <jinxer-wm>	 Deployment qwen36-27b-predictor-00011-deployment in experimental at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=experimental&var-deployment=qwen36-27b-predictor-00011-deployment - ...
[08:58:55] <jinxer-wm>	 https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[09:08:49] <jinxer-wm>	 RESOLVED: KubernetesDeploymentUnavailableReplicas: ...
[09:08:49] <jinxer-wm>	 Deployment qwen36-27b-predictor-00011-deployment in experimental at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=experimental&var-deployment=qwen36-27b-predictor-00011-deployment - ...
[09:08:49] <jinxer-wm>	 https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[10:04:20] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06Traffic, 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12064615 (10Clement_Goubert)
[10:21:56] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): LLM observability - https://phabricator.wikimedia.org/T429597#12064714 (10isarantopoulos)
[10:22:00] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 05Goal: Setup MI300X nodes for LLM serving - https://phabricator.wikimedia.org/T424322#12064715 (10isarantopoulos)
[12:13:09] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06Traffic, 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12065080 (10isarantopoulos) Thanks for all the work here!  Regarding the OOM errors: since we’re using 2x24 GB partitions, shouldn’t the...
[13:33:17] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06Traffic, 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12065527 (10gkyziridis) ==== Update ====  Qwen3.6-27B-FP8 KServe Deployment — Blocking Issue The Problem I parsed is that even the logs o...
[13:59:02] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26): Migrate TTS inference logic from v0 to v1 (without orchestration, storage, and serving layers) - https://phabricator.wikimedia.org/T430536 (10kevinbazira) 03NEW
[14:04:48] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12065889 (10isarantopoulos) I'm leaning towards option 1 as well. With the current status and the fact that we are starting to deploy services on these hosts...
[14:06:52] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: `mediawiki.page_change.v1`: negative namespace_id schema validation errors - https://phabricator.wikimedia.org/T421237#12065906 (10Ottomata) From @gmodena in [[ https://wikimedia.slack.com/archive...
[14:07:20] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: `mediawiki.page_change.v1`: negative namespace_id schema validation errors - https://phabricator.wikimedia.org/T421237#12065916 (10Ottomata)
[14:46:10] <wikibugs>	 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26): Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12066235 (10OKarakaya-WMF) I've deployed the applications in the following patches to staging: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1305889 https:...
[16:57:37] <wikibugs>	 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: `mediawiki.page_change.v1`: negative namespace_id schema validation errors - https://phabricator.wikimedia.org/T421237#12067154 (10Ottomata) FYI, I merged [Add first_edit_dt to UserEntitySerialize...