[10:30:12] (03PS1) 10VadymTS1: Add Ukrainian translation for ORES extension special page [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1295868 (https://phabricator.wikimedia.org/T427713) [10:34:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [10:34:49] Deployment recommendation-api-ng-main in recommendation-api-ng at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=recommendation-api-ng&var-deployment=recommendation-api-ng-main - ... [10:34:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [10:39:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [10:39:49] Deployment recommendation-api-ng-main in recommendation-api-ng at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=recommendation-api-ng&var-deployment=recommendation-api-ng-main - ... [10:39:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [12:01:42] (03PS1) 10Ozge: Add editing-suggestions KServe model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1295879 [12:01:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [12:01:49] Deployment recommendation-api-ng-main in recommendation-api-ng at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=recommendation-api-ng&var-deployment=recommendation-api-ng-main - ... [12:01:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [12:02:36] (03PS2) 10Ozge: Add editing-suggestions KServe model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1295879 [12:05:18] 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - api - https://phabricator.wikimedia.org/T427794 (10OKarakaya-WMF) 03NEW [12:05:24] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971733 (10ops-monitoring-bot) Draining ganeti2027.codfw.wmnet of running VMs [12:06:40] (03PS3) 10Ozge: Add editing-suggestions KServe model server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1295879 (https://phabricator.wikimedia.org/T427794) [12:07:14] 06Machine-Learning-Team (Q4 FY2025-26): Editing Suggestions - api - https://phabricator.wikimedia.org/T427794#11971735 (10OKarakaya-WMF) https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1295879 [12:08:34] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971736 (10ops-monitoring-bot) VM kubestagemaster2005.codfw.wmnet switching disk type to drbd [12:21:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [12:21:49] Deployment recommendation-api-ng-main in recommendation-api-ng at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=recommendation-api-ng&var-deployment=recommendation-api-ng-main - ... [12:21:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [12:23:51] (03CR) 10AikoChou: [C:03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1295361 (https://phabricator.wikimedia.org/T418493) (owner: 10Bartosz Wójtowicz) [12:26:26] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971797 (10ops-monitoring-bot) Draining ganeti2027.codfw.wmnet of running VMs [12:27:29] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971798 (10ops-monitoring-bot) VM kubestagemaster2005.codfw.wmnet switching disk type to plain [12:29:14] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971800 (10ops-monitoring-bot) Draining ganeti2027.codfw.wmnet of running VMs [12:32:26] (03CR) 10Bartosz Wójtowicz: [C:03+2] outlink-topic-model: Sanitize output from V2 endpoint. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1295361 (https://phabricator.wikimedia.org/T418493) (owner: 10Bartosz Wójtowicz) [12:36:59] (03Merged) 10jenkins-bot: outlink-topic-model: Sanitize output from V2 endpoint. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1295361 (https://phabricator.wikimedia.org/T418493) (owner: 10Bartosz Wójtowicz) [13:16:08] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Configure LiftWing's openAPI specs into mediawiki-config - https://phabricator.wikimedia.org/T426081#11972011 (10Clement_Goubert) >>! In T426081#11960139, @Mooeypoo wrote: >>>! In T426081#11935377, @gkyziridis wrote: >>>>! In T426081#11... [13:21:36] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Configure LiftWing's openAPI specs into mediawiki-config - https://phabricator.wikimedia.org/T426081#11972039 (10HCoplin-WMF) Piling on a bit, but want to emphasize that we definitely need an endpoint to pull this information. As @Mooey... [13:27:58] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Configure LiftWing's openAPI specs into mediawiki-config - https://phabricator.wikimedia.org/T426081#11972057 (10Clement_Goubert) >>! In T426081#11972039, @HCoplin-WMF wrote: ... > My general preference matches @Clement_Goubert's sugges... [13:47:23] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Configure LiftWing's openAPI specs into mediawiki-config - https://phabricator.wikimedia.org/T426081#11972213 (10gkyziridis) Thank you all for your advices and comments. It is much appreciated! I would like to add some clarification to... [15:32:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [15:32:49] Deployment revertrisk-multilingual-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ... [15:32:49] https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-multilingual-predictor-00001-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [15:37:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [15:37:49] Deployment revertrisk-multilingual-predictor-00001-deployment in revertrisk at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - ... [15:37:49] https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revertrisk&var-deployment=revertrisk-multilingual-predictor-00001-deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:03:30] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973104 (10ops-monitoring-bot) Draining ganeti2045.codfw.wmnet of running VMs [16:05:38] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973112 (10ops-monitoring-bot) VM aux-k8s-etcd2003.codfw.wmnet switching disk type to drbd [16:15:24] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973137 (10MoritzMuehlenhoff) [16:38:14] 06Machine-Learning-Team (Q4 FY2025-26): Add word-level timestamps in TTS prototype - https://phabricator.wikimedia.org/T427488#11973196 (10kevinbazira) We have added word-level timestamps to the TTS prototype. Each audio section (.mp3) now comes with a companion WebVTT caption file (.vtt) with per-word start and... [16:44:40] 06Machine-Learning-Team (Q4 FY2025-26): Add word-level timestamps in TTS prototype - https://phabricator.wikimedia.org/T427488#11973212 (10kevinbazira) We have also added a demo of this feature in the TTS prototype UI. Now when you play a section's audio, the transcript below the audio player highlights each spo... [17:00:40] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973321 (10ops-monitoring-bot) VM dse-k8s-etcd2001.codfw.wmnet switching disk type to drbd [17:26:50] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06collaboration-services, 10ServiceOps-Mediawiki, and 2 others: Configure LiftWing's openAPI specs into mediawiki-config - https://phabricator.wikimedia.org/T426081#11973429 (10Clement_Goubert) I do object to using `mediawiki-config` static files in this... [17:59:29] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973526 (10ops-monitoring-bot) Draining ganeti2045.codfw.wmnet of running VMs [18:01:38] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973540 (10ops-monitoring-bot) VM aux-k8s-etcd2003.codfw.wmnet switching disk type to plain [18:03:04] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973544 (10ops-monitoring-bot) VM dse-k8s-etcd2001.codfw.wmnet switching disk type to plain [18:05:53] 06Machine-Learning-Team, 06Discovery-Search, 06Infrastructure-Foundations, 10netops, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973560 (10ops-monitoring-bot) Draining ganeti2045.codfw.wmnet of running VMs [19:01:31] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 06collaboration-services, 10ServiceOps-Mediawiki, and 2 others: Configure LiftWing's openAPI specs into mediawiki-config - https://phabricator.wikimedia.org/T426081#11973769 (10HCoplin-WMF) Also want to chime in that I'm opposed to hosting these files in... [19:26:37] (03CR) 10AikoChou: "Thanks for working on this! I left some comments." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1295879 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge)