[02:09:25] 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: `mediawiki.page_change.v1`: negative namespace_id schema validation errors - https://phabricator.wikimedia.org/T421237#12078708 (10Ottomata) [06:50:17] (03PS1) 10Kevin Bazira: tts: Add TTS service inference layer [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306880 (https://phabricator.wikimedia.org/T430536) [07:09:28] (03PS1) 10Kevin Bazira: test: Add TTS model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307057 (https://phabricator.wikimedia.org/T430536) [07:10:57] (03CR) 10CI reject: [V:04-1] test: Add TTS model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307057 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [07:15:16] (03PS2) 10Kevin Bazira: test: Add TTS model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307057 (https://phabricator.wikimedia.org/T430536) [08:05:48] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26): Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12079194 (10OKarakaya-WMF) Deployed to prod: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1306930 [08:16:41] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26): Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12079268 (10OKarakaya-WMF) I see apps (article-country, revscoring, revertrisk-multilingual, revertrisk-wikidata) are up-to-date on prod deployment charts now. @achou , @... [08:33:18] (03CR) 10Bartosz Wójtowicz: "Thank you for the work! I managed to spin up the server and test that it's working! Leaving a couple of small comments :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306880 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:50:15] 06Machine-Learning-Team (Q4 FY2025-26): LLM observability - https://phabricator.wikimedia.org/T429597#12079412 (10DPogorzelski-WMF) @kevinbazira some graphs don't have data at the moment so you can ignore those but could you otherwise do a pass on the vllm related graphs and see whether they make sense? https://... [09:12:33] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918 (10ayounsi) 03NEW p:05Triage→03Medium The #Cloud-Services project tag is not intended to have any tasks. Please check the list... [09:20:43] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922 (10ayounsi) 03NEW [09:22:12] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12079601 (10ayounsi) @cmooney @Papaul can one of you take that upgrade ? And update the date/time to anything that suits you in that week. [09:22:24] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079602 (10ayounsi) @cmooney @Papaul can one of you take that upgrade ? And update the date/time to anything that suits you in that... [09:25:03] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079609 (10CWilliams-WMF) [09:25:31] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079614 (10CWilliams-WMF) [09:27:43] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12079634 (10ayounsi) [09:28:03] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079636 (10ayounsi) [09:30:44] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079648 (10CWilliams-WMF) [09:33:57] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12079679 (10CWilliams-WMF) [09:34:34] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12079690 (10CWilliams-WMF) [09:39:25] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12079747 (10Marostegui) @ayounsi I'll be on holidays most likely when this happens, for db2162 @CWilliams-WMF is switching it at T430925 pc2022... [09:42:22] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079752 (10Marostegui) db2160 needs just a downtime. dbproxy will fail during the maintenance (just irc notification) and they'll re... [09:46:02] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12079779 (10Marostegui) [09:46:32] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06Infrastructure-Foundations, and 2 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079781 (10ayounsi) [09:47:54] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079785 (10ayounsi) [09:48:03] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079787 (10Marostegui) [09:48:50] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079788 (10ayounsi) [09:49:04] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079789 (10ayounsi) [09:57:53] 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Multi-language support - https://phabricator.wikimedia.org/T424988#12079821 (10OKarakaya-WMF) closing this ticket as multilanguage support will be handled in a new goal. [09:59:16] 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Multi-language support - https://phabricator.wikimedia.org/T424988#12079833 (10OKarakaya-WMF) 05Open→03Resolved [10:04:13] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079891 (10cmooney) a:03cmooney [10:04:20] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12079892 (10cmooney) a:03cmooney [10:04:40] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079893 (10cmooney) a:03cmooney [10:06:12] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079906 (10cmooney) [10:09:11] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079912 (10jcrespo) [10:09:16] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12079913 (10cmooney) [10:10:47] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079920 (10jcrespo) backup2012: it is repo backups, should be ok for a small downtime, at most 1 backup run will fail, it will be re... [10:11:51] 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Eval pipeline for enwiki - https://phabricator.wikimedia.org/T425957#12079923 (10OKarakaya-WMF) pipeline steps are implemented as script and more information added to a document: https://gitlab.wikimedia.org/repos/machine-learning/exploratory-noteboo... [10:12:43] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079927 (10cmooney) [10:13:34] 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - Eval pipeline for enwiki - https://phabricator.wikimedia.org/T425957#12079930 (10OKarakaya-WMF) 05Open→03Resolved [11:53:03] 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: Access control for LiftWing LLM services exposed to external clients through REST Gateway - https://phabricator.wikimedia.org/T426749#12080179 (10BWojtowicz-WMF) The changes to the gateway (https://gerrit.wikimedia.org/r/c/op... [11:53:15] 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 10ServiceOps-SharedInfra: Access control for LiftWing LLM services exposed to external clients through REST Gateway - https://phabricator.wikimedia.org/T426749#12080181 (10BWojtowicz-WMF) 05Open→03Resolved [12:26:23] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12080486 (10cmooney) [12:27:05] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12080487 (10cmooney) [13:41:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [13:41:54] Deployment calico-kube-controllers in kube-system at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=kube-system&var-deployment=calico-kube-controllers - ... [13:41:54] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [13:49:11] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12080897 (10CWilliams-WMF) [13:57:14] 06Machine-Learning-Team, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, 06ServiceOps new: codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12080968 (10CWilliams-WMF) [14:00:55] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12080994 (10CWilliams-WMF) [14:02:22] (03PS3) 10Kevin Bazira: test: Add TTS model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307057 (https://phabricator.wikimedia.org/T430536) [14:06:01] (03PS2) 10Kevin Bazira: tts: Add TTS service inference layer [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306880 (https://phabricator.wikimedia.org/T430536) [14:24:56] (03CR) 10Kevin Bazira: tts: Add TTS service inference layer (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1306880 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [14:47:59] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12081259 (10Gehel) [14:48:12] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12081261 (10Gehel) [14:57:19] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12081351 (10Gehel) [16:08:21] 06Machine-Learning-Team, 10Automoderator, 06Moderator-Tools-Team: Use multilingual revert risk model in Automoderator on supported wikis - https://phabricator.wikimedia.org/T365581#12081846 (10Samwalton9-WMF) 05Open→03Resolved a:03Samwalton9-WMF [16:10:29] 06Machine-Learning-Team (Q4 FY2025-26), 06ServiceOps new, 13Patch-For-Review: Update prod kserve/knative - https://phabricator.wikimedia.org/T426823#12081862 (10DPogorzelski-WMF) 05Open→03Resolved Done [16:36:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [16:36:49] Deployment calico-kube-controllers in kube-system at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=kube-system&var-deployment=calico-kube-controllers - ... [16:36:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:37:19] FIRING: KubernetesDeploymentUnavailableReplicas: ... [16:37:19] Deployment calico-kube-controllers in kube-system at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=kube-system&var-deployment=calico-kube-controllers - ... [16:37:22] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:52:19] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [16:52:19] Deployment calico-kube-controllers in kube-system at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=kube-system&var-deployment=calico-kube-controllers - ... [16:52:22] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [18:09:47] 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017 (10isarantopoulos) 03NEW [18:18:41] 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12082566 (10isarantopoulos) p:05Triage→03High [18:32:12] 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12082620 (10isarantopoulos) The deployment of qwen3.6 model might have affected this https://phabricator.wikimedia.org/T425680#12075549 [18:41:07] 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12082662 (10isarantopoulos) I have manually deleted some pods and a llm-qwen3-14b service that was deployed recently and was schedule on the same node htt... [18:42:33] 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12082671 (10isarantopoulos) The cope-b service seems to work now, but we need to monitor in case pods are evicted due to the same reason.