[04:49:27] (03PS1) 10Kevin Bazira: cope-b-a4b: Enable vLLM metrics [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307668 (https://phabricator.wikimedia.org/T431136) [05:21:35] (03PS1) 10Kevin Bazira: tts: containerize model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307674 (https://phabricator.wikimedia.org/T430536) [05:26:15] (03CR) 10CI reject: [V:04-1] tts: containerize model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307674 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [05:31:00] (03PS2) 10Kevin Bazira: tts: containerize model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307674 (https://phabricator.wikimedia.org/T430536) [05:35:18] (03CR) 10CI reject: [V:04-1] tts: containerize model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307674 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [05:41:41] (03CR) 10Kevin Bazira: "The 3 failing tests (editing-suggestions, revise-tone-task-generator, and edit-check) will be fixed in:" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307674 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [07:34:13] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26): Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12088798 (10OKarakaya-WMF) Thank you Kevin, I'm looking into editing-suggestions. [07:35:21] (03CR) 10Dpogorzelski: [C:03+1] cope-b-a4b: Enable vLLM metrics [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307668 (https://phabricator.wikimedia.org/T431136) (owner: 10Kevin Bazira) [07:35:50] (03CR) 10Kevin Bazira: [C:03+2] cope-b-a4b: Enable vLLM metrics [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307668 (https://phabricator.wikimedia.org/T431136) (owner: 10Kevin Bazira) [07:37:40] (03Merged) 10jenkins-bot: cope-b-a4b: Enable vLLM metrics [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307668 (https://phabricator.wikimedia.org/T431136) (owner: 10Kevin Bazira) [07:40:26] (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you, LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307674 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [07:48:45] 10Lift-Wing, 06Machine-Learning-Team: Generate OpenAPI descriptions for Lift Wing APIs - https://phabricator.wikimedia.org/T419455#12088846 (10isarantopoulos) [07:48:54] 06Machine-Learning-Team: Editing Suggestions - Replace reference [ and ] in target with a special character. - https://phabricator.wikimedia.org/T430812#12088849 (10isarantopoulos) [07:48:55] 06Machine-Learning-Team: Editing Suggestions - Update static descriptions. - https://phabricator.wikimedia.org/T430030#12088851 (10isarantopoulos) [07:48:57] 06Machine-Learning-Team: Editing Suggestions - Updating documentation - https://phabricator.wikimedia.org/T430029#12088853 (10isarantopoulos) [07:49:00] 10Lift-Wing, 06Machine-Learning-Team, 07Essential-Work: Rebuild vLLM base images to use deb.debian.org in apt sources - https://phabricator.wikimedia.org/T429667#12088855 (10isarantopoulos) [07:49:04] 06Machine-Learning-Team: Editing Suggestions - Phase I dataset - https://phabricator.wikimedia.org/T429470#12088857 (10isarantopoulos) [07:49:06] 06Machine-Learning-Team: LLM Generated Editing Suggestions - API changes. - https://phabricator.wikimedia.org/T428740#12088865 (10isarantopoulos) [07:49:10] 06Machine-Learning-Team: Editing Suggestions - Increase/refresh LLM generated suggestions. - https://phabricator.wikimedia.org/T428882#12088863 (10isarantopoulos) [07:49:14] 06Machine-Learning-Team, 06SRE, 10SRE-Access-Requests: Requesting access to ml-lab-users for mfossati - https://phabricator.wikimedia.org/T429148#12088859 (10isarantopoulos) [07:49:20] 10Lift-Wing, 06Machine-Learning-Team, 07Essential-Work: Update WMF Debian vLLM image to use pre-built wheels from upstream - https://phabricator.wikimedia.org/T428577#12088867 (10isarantopoulos) [07:49:25] 06Machine-Learning-Team, 07OKR-Work: Improve and optimize Article Topics model to better handle queries with `revision_id` - https://phabricator.wikimedia.org/T428127#12088871 (10isarantopoulos) [07:49:29] 06Machine-Learning-Team, 10Add-Link-Structured-Task, 07Essential-Work, 07Wikimedia-production-error: MentionRegexException in link-recommendation service - https://phabricator.wikimedia.org/T420255#12088869 (10isarantopoulos) [07:49:33] 10Lift-Wing, 06Machine-Learning-Team: Deploy CoPE-B-A4B on LiftWing - https://phabricator.wikimedia.org/T427497#12088878 (10isarantopoulos) [07:49:41] 06Machine-Learning-Team, 13Patch-For-Review: Editing Suggestions - api - https://phabricator.wikimedia.org/T427794#12088876 (10isarantopoulos) [07:49:45] 06Machine-Learning-Team: Add word-level timestamps in TTS prototype - https://phabricator.wikimedia.org/T427488#12088880 (10isarantopoulos) [07:49:49] 10Lift-Wing, 06Machine-Learning-Team, 06ServiceOps new, 10ServiceOps-SharedInfra, 13Patch-For-Review: Expose LiftWing API for serving the openapi-specs through the /docs yaml files. - https://phabricator.wikimedia.org/T427902#12088873 (10isarantopoulos) [07:49:57] 06Machine-Learning-Team, 06Data-Engineering: EventStreams - Expose mediawiki.page_outlink_topic_prediction_change.v1 stream - https://phabricator.wikimedia.org/T427416#12088882 (10isarantopoulos) [07:50:01] 06Machine-Learning-Team: Add section heading announcements in TTS prototype - https://phabricator.wikimedia.org/T427173#12088886 (10isarantopoulos) [07:50:05] 06Machine-Learning-Team: Add static audio URL for native OS audio players in TTS prototype - https://phabricator.wikimedia.org/T427262#12088884 (10isarantopoulos) [07:50:09] 06Machine-Learning-Team, 06ServiceOps new: Update prod kserve/knative - https://phabricator.wikimedia.org/T426823#12088890 (10isarantopoulos) [07:50:16] 06Machine-Learning-Team, 10Ceph, 06Infrastructure-Foundations, 10SRE-swift-storage: Move the Docker Registry's /ml prefix to S3/apus - https://phabricator.wikimedia.org/T420978#12088888 (10isarantopoulos) [07:50:26] 10Lift-Wing, 06Machine-Learning-Team: prediction_classification events rejected by EventGate due to extra schema properties - https://phabricator.wikimedia.org/T426807#12088894 (10isarantopoulos) [07:50:32] 06Machine-Learning-Team, 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#12088892 (10isarantopoulos) [07:50:36] 06Machine-Learning-Team: Fix text normalization edge cases in TTS prototype - https://phabricator.wikimedia.org/T426756#12088900 (10isarantopoulos) [07:50:42] 06Machine-Learning-Team, 13Patch-For-Review: Experiment with new kserve version on ml-staging-codfw - https://phabricator.wikimedia.org/T419722#12088896 (10isarantopoulos) [07:50:46] 06Machine-Learning-Team, 06ServiceOps new, 10ServiceOps-SharedInfra: Access control for LiftWing LLM services exposed to external clients through REST Gateway - https://phabricator.wikimedia.org/T426749#12088902 (10isarantopoulos) [07:50:52] 10Lift-Wing, 06Machine-Learning-Team, 13Patch-For-Review: Upgrade production vLLM image to use vLLM version >= 0.19 - https://phabricator.wikimedia.org/T426766#12088898 (10isarantopoulos) [07:50:56] 10Lift-Wing, 06Machine-Learning-Team, 06collaboration-services, 10ServiceOps-Mediawiki, 10ServiceOps-SharedInfra: Configure LiftWing's openAPI specs into mediawiki-config - https://phabricator.wikimedia.org/T426081#12088904 (10isarantopoulos) [07:51:02] 06Machine-Learning-Team: Editing Suggestions - Multi-language support - https://phabricator.wikimedia.org/T424988#12088910 (10isarantopoulos) [07:51:06] 06Machine-Learning-Team: Editing Suggestions - Eval pipeline for enwiki - https://phabricator.wikimedia.org/T425957#12088906 (10isarantopoulos) [07:51:10] 06Machine-Learning-Team: Editing Suggestions - Dataset generation logic from edits - https://phabricator.wikimedia.org/T424347#12088916 (10isarantopoulos) [07:51:14] 06Machine-Learning-Team: Explore options to run TTS models for evaluation - https://phabricator.wikimedia.org/T424378#12088914 (10isarantopoulos) [07:51:20] 06Machine-Learning-Team, 10Toolforge (Quota-requests): Request increased quota for wiki-tts Toolforge tool - https://phabricator.wikimedia.org/T425804#12088908 (10isarantopoulos) [07:51:24] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: MI300 machines need startup tweaks - https://phabricator.wikimedia.org/T420507#12088912 (10isarantopoulos) [07:51:28] 06Machine-Learning-Team, 06ServiceOps new, 06Traffic: k8s changes needed to allow article topic (and other future isvcs) to use the kserve v2 inference protocol (and gRPC) - https://phabricator.wikimedia.org/T424049#12088920 (10isarantopoulos) [07:51:37] 06Machine-Learning-Team, 07Essential-Work: Alerts about the article quality model - https://phabricator.wikimedia.org/T424018#12088922 (10isarantopoulos) [07:51:41] 10Lift-Wing, 06Machine-Learning-Team, 06SRE: Fix securityContext propagation in liftwing - https://phabricator.wikimedia.org/T423149#12088924 (10isarantopoulos) [07:51:47] 06Machine-Learning-Team, 07OKR-Work: Integrate Article Topic model with the new caching service - https://phabricator.wikimedia.org/T418493#12088932 (10isarantopoulos) [07:51:51] 06Machine-Learning-Team, 07OKR-Work: Add 'iommu=pt' kernel parameter on MI300x nodes for direct GPU-to-GPU communication (PCIe P2P) - https://phabricator.wikimedia.org/T421461#12088934 (10isarantopoulos) [07:51:55] 06Machine-Learning-Team, 07Essential-Work, 07sre-alert-triage: Alert in need of triage: HelmfileAdminNGPendingChanges (instance deploy1003:9100) - https://phabricator.wikimedia.org/T414971#12088936 (10isarantopoulos) [07:52:01] 06Machine-Learning-Team, 07OKR-Work, 13Patch-For-Review: Modify article topic model code to support kserve v2 inference protocol - https://phabricator.wikimedia.org/T423582#12088930 (10isarantopoulos) [07:52:05] 06Machine-Learning-Team: Reduce logstash logs from machine learning infra - https://phabricator.wikimedia.org/T416384#12088940 (10isarantopoulos) [07:52:11] 06Machine-Learning-Team, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06ServiceOps new, 07Kubernetes: Set cert-manager leader election namespace to cert-manager - https://phabricator.wikimedia.org/T383553#12088938 (10isarantopoulos) [07:52:17] 06Machine-Learning-Team, 07Essential-Work: edit-check isvc endpoint returns plain text errors instead of JSON on invalid requests - https://phabricator.wikimedia.org/T422482#12088944 (10isarantopoulos) [07:52:21] 06Machine-Learning-Team, 13Patch-For-Review: Investigate how to enable the swagger UI for InferenceService resources - https://phabricator.wikimedia.org/T332602#12088942 (10isarantopoulos) [07:52:25] 06Machine-Learning-Team: Edit suggestions - Precision recall on grammar edit suggestions based on edits. - https://phabricator.wikimedia.org/T421672#12088946 (10isarantopoulos) [07:52:29] 06Machine-Learning-Team: Edit Suggestions - Edit suggestion generation with pre-defined edit types - https://phabricator.wikimedia.org/T418102#12088950 (10isarantopoulos) [07:52:33] 06Machine-Learning-Team, 07OKR-Work, 13Patch-For-Review: Deploy gpt-oss-safeguard-20b on LiftWing - https://phabricator.wikimedia.org/T418350#12088948 (10isarantopoulos) [07:53:12] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26): Investigate enabling gRPC in LiftWing model servers - https://phabricator.wikimedia.org/T421903#12088956 (10isarantopoulos) 05Open→03Resolved gRPC has been enabled for outlink articletopic model and the documentation for this work exists in https://w... [07:54:24] 10Lift-Wing, 06Machine-Learning-Team: Investigate enabling gRPC in LiftWing model servers - https://phabricator.wikimedia.org/T421903#12088960 (10isarantopoulos) [07:57:08] 06Machine-Learning-Team, 10Cloud-VPS (Project-requests): Request creation of wikitts VPS project - https://phabricator.wikimedia.org/T425909#12088965 (10isarantopoulos) [08:13:30] (03PS1) 10Ozge: editing_suggestions: Bump bookworm base image to 20260705 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307700 (https://phabricator.wikimedia.org/T429675) [08:15:23] (03PS2) 10Ozge: editing_suggestions: Bump bookworm base image to 20260705 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307700 (https://phabricator.wikimedia.org/T429675) [08:15:58] (03PS3) 10Ozge: editing_suggestions: Bump bookworm base image to 20260705 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307700 (https://phabricator.wikimedia.org/T429675) [08:16:07] (03PS4) 10Ozge: editing_suggestions: Bump bookworm base image to 20260705 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307700 (https://phabricator.wikimedia.org/T429675) [08:16:15] (03PS5) 10Ozge: editing_suggestions: Bump bookworm base image to 20260705 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307700 (https://phabricator.wikimedia.org/T429675) [08:23:51] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12089053 (10OKarakaya-WMF) created a [patch](https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1307700) for editing sugges... [08:27:27] (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307700 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [08:28:38] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 10Semantic Search, 06Traffic, 07Essential-Work: Transparent DNS Routing for LiftWing Services (eqiad vs Multi-DC) - https://phabricator.wikimedia.org/T422253#12089070 (10DPogorzelski-WMF) @isarantopoulos I think I agree with @elukey here and currently... [08:29:35] (03CR) 10Ozge: [C:03+2] editing_suggestions: Bump bookworm base image to 20260705 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307700 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [08:29:43] (03CR) 10Kevin Bazira: [C:03+1] editing_suggestions: Bump bookworm base image to 20260705 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307700 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [08:33:10] (03Merged) 10jenkins-bot: editing_suggestions: Bump bookworm base image to 20260705 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307700 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [08:50:20] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 10Semantic Search, 06Traffic, 07Essential-Work: Transparent DNS Routing for LiftWing Services (eqiad vs Multi-DC) - https://phabricator.wikimedia.org/T422253#12089169 (10isarantopoulos) 05Open→03Resolved Sounds good. I'll be resolving this task... [09:31:40] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: MI300 machines need startup tweaks - https://phabricator.wikimedia.org/T420507#12089316 (10elukey) >>! In T420507#11726566, @elukey wrote: > Worth to remember - amd-smi is currently relying on a manual fix in the Python files due to https://g... [09:49:37] 06Machine-Learning-Team (Q4 FY2025-26), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12089435 (10Gehel) 05Open→03In progress [09:49:38] 06Machine-Learning-Team (Q4 FY2025-26), 10Liberica, 10Prod-Kubernetes, 06Traffic, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12089434 (10Gehel) [09:51:15] 06Machine-Learning-Team, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12089438 (10Gehel) [09:52:36] 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12089442 (10Gehel) [09:52:37] 06Machine-Learning-Team (Q4 FY2025-26), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12089443 (10Gehel) 05Open→03In progress [09:56:24] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 10Semantic Search, 06Traffic, 07Essential-Work: Transparent DNS Routing for LiftWing Services (eqiad vs Multi-DC) - https://phabricator.wikimedia.org/T422253#12089463 (10achou) +1. I suggest documenting this on the [[ https://wikitech.wikimedia.org... [10:25:29] 06Machine-Learning-Team, 10decommission-hardware, 13Patch-For-Review: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12089618 (10BTullis) Can we hold off de-racking these for the time being, please? I'm just looking into something and they might be useful, even for a little while. [10:27:28] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12089620 (10CWilliams-WMF) > can you support/be around for this? Yes, at least one of us will be around [10:28:17] 06Machine-Learning-Team (Q4 FY2025-26): Qwen3-14b responses are corrupted when running on 2x24GB partitions of MI300x - https://phabricator.wikimedia.org/T431268 (10BWojtowicz-WMF) 03NEW [10:29:06] 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Redeploy Liftwing pick up current mirror - https://phabricator.wikimedia.org/T429675#12089641 (10OKarakaya-WMF) updated editing suggestions on both staging and production. [10:29:45] 06Machine-Learning-Team (Q4 FY2025-26): Qwen3-14b responses are corrupted when running on 2x24GB partitions of MI300x - https://phabricator.wikimedia.org/T431268#12089642 (10BWojtowicz-WMF) a:03BWojtowicz-WMF [10:32:23] 06Machine-Learning-Team (Q4 FY2025-26): Qwen3-14b responses are corrupted when running on 2x24GB partitions of MI300x - https://phabricator.wikimedia.org/T431268#12089655 (10isarantopoulos) This seems related to the qwen3.6-27b deployment attempt on multiple partitions in https://phabricator.wikimedia.org/T42568... [10:37:42] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Qwen3-14b responses are corrupted when running on 2x24GB partitions of MI300x - https://phabricator.wikimedia.org/T431268#12089681 (10BWojtowicz-WMF) One potential issue I've noticed in startup logs is this: ` (EngineCore_DP0 pid=186) [aiter] WARNING:... [10:57:45] (03CR) 10Kevin Bazira: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307674 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [11:06:30] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Qwen3-14b responses are corrupted when running on 2x24GB partitions of MI300x - https://phabricator.wikimedia.org/T431268#12089791 (10BWojtowicz-WMF) Moved the experimental qwen3-14b deployment to a single 192GB GPU on ml-serve1012 and everything work... [11:07:35] (03CR) 10Kevin Bazira: [V:03+2 C:03+2] tts: containerize model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307674 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [11:12:48] (03CR) 10CI reject: [V:04-1] tts: containerize model-server unit tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1307674 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [11:39:20] 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Add Wikidata RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T420883#12089892 (10achou) > Is there a reason this needs its own special stream? other page change streams have all wikis in the s... [11:59:17] 10Lift-Wing, 06Machine-Learning-Team, 10Semantic Search, 06Traffic, 07Essential-Work: Transparent DNS Routing for LiftWing Services (eqiad vs Multi-DC) - https://phabricator.wikimedia.org/T422253#12090009 (10isarantopoulos) [12:03:54] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Enable vLLM Prometheus metrics for LiftWing LLM inference services - https://phabricator.wikimedia.org/T431136#12090029 (10kevinbazira) The cope-b-a4b isvc now emits vLLM metrics as shown below: {P94732} [12:05:06] 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Enable vLLM Prometheus metrics for LiftWing LLM inference services - https://phabricator.wikimedia.org/T431136#12090030 (10kevinbazira) [12:05:36] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 07Essential-Work: Move Lab machines into analytics net for DL access and switch to homedirs on Ceph - https://phabricator.wikimedia.org/T380279#12090031 (10isarantopoulos) [12:05:58] 06Machine-Learning-Team (Q1 FY2026-27): Possibility to transition ml-serve[2001-2008] and and ml-staging[2001-2002] to 10G - https://phabricator.wikimedia.org/T378714#12090032 (10isarantopoulos) [12:07:10] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: Add the ML team to the POSIX group `docker` on the ML lab machines. - https://phabricator.wikimedia.org/T393566#12090034 (10isarantopoulos) [12:07:51] 06Machine-Learning-Team (Q1 FY2026-27), 10decommission-hardware, 13Patch-For-Review: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12090036 (10isarantopoulos) [12:08:02] 06Machine-Learning-Team (Q1 FY2026-27): Build the debian packages for amd node labeller and gpu plugin to pick up the new changes - https://phabricator.wikimedia.org/T423459#12090037 (10isarantopoulos) [12:08:12] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12090038 (10isarantopoulos) [12:09:22] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: ml-lab: Update pytorch-rocm to a current ROCm-compatible release - https://phabricator.wikimedia.org/T428023#12090059 (10isarantopoulos) [12:09:25] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: Possibility to transition ml-serve[2001-2008] and and ml-staging[2001-2002] to 10G - https://phabricator.wikimedia.org/T378714#12090058 (10isarantopoulos) [12:09:26] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: Build the debian packages for amd node labeller and gpu plugin to pick up the new changes - https://phabricator.wikimedia.org/T423459#12090060 (10isarantopoulos) [12:09:30] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: Machine Learning: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421721#12090061 (10isarantopoulos) [12:09:36] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 07sre-alert-triage: Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969#12090062 (10isarantopoulos) [12:09:44] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 07Essential-Work: Support GPU Partition specification in deployments - https://phabricator.wikimedia.org/T424923#12090064 (10isarantopoulos) [12:09:51] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: monitoring: View GPU usage per LLM deployment/model - https://phabricator.wikimedia.org/T429236#12090065 (10isarantopoulos) [12:09:57] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 10decommission-hardware, 13Patch-For-Review: decommission ml-cache* - https://phabricator.wikimedia.org/T430654#12090063 (10isarantopoulos) [12:10:03] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: monitoring: Grafana dashboard for LLM serving on MI300X - https://phabricator.wikimedia.org/T429237#12090066 (10isarantopoulos) [12:10:09] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: LLM observability - https://phabricator.wikimedia.org/T429597#12090067 (10isarantopoulos) [12:10:15] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 13Patch-For-Review: Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12090068 (10isarantopoulos) [12:10:21] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 10Liberica, 10Prod-Kubernetes, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12090069 (10isarantopoulos) [12:12:30] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE: LLM observability - https://phabricator.wikimedia.org/T429597#12090072 (10kevinbazira) Following T429597#12086389, we enabled metrics emission in one isvc (cope-b-a4b). This service now emits vLLM metrics as shown in: T431136#12090029 We'll do the s... [12:30:34] 06Machine-Learning-Team, 07Essential-Work, 13Patch-For-Review: MI300 machines need startup tweaks - https://phabricator.wikimedia.org/T420507#12090140 (10elukey) 05Resolved→03Open Didn't realize this was closed. Reopening it to complete the above :) [13:35:16] 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE, 10Liberica, 10Prod-Kubernetes, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12090498 (10ops-monitoring-bot) Roll-reboot of nodes in ml-serve-codfw cluster started by klausman: * ml-serve-... [14:28:26] 06Machine-Learning-Team, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12090701 (10cmooney) p:05Triage→03Medium [15:39:48] 06Machine-Learning-Team (Q1 FY2026-27), 05Goal, 07OKR-Work: Q2 FY2025-26 Goal: Generate a list of edit suggestions using machine learning - https://phabricator.wikimedia.org/T409863#12090999 (10OKarakaya-WMF) hey @Sucheta-Salgaonkar-WMF , we have finished the tasks on MLTeam side for this goal. I see there a... [16:52:20] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12091218 (10Dzahn) [16:53:54] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12091221 (10Dzahn) re: contint2002 - This is not the main CI server, that is currently contint1002, but it is still used to a certain... [17:08:29] 10Lift-Wing, 06Machine-Learning-Team, 10Semantic Search, 06Traffic, 07Essential-Work: Transparent DNS Routing for LiftWing Services (eqiad vs Multi-DC) - https://phabricator.wikimedia.org/T422253#12091255 (10dcausse) Thanks for this looking, so I understand that for internal usecases (e.g. qwen3-embe... [17:58:59] 10Lift-Wing, 06Machine-Learning-Team, 10Semantic Search, 06Traffic, 07Essential-Work: Transparent DNS Routing for LiftWing Services (eqiad vs Multi-DC) - https://phabricator.wikimedia.org/T422253#12091490 (10isarantopoulos) Hey @dcausse! yes that is correct. [20:06:02] 06Machine-Learning-Team, 06App Experience: Articles on men are appearing as suggestions to users based on a stated interested in women biographies - https://phabricator.wikimedia.org/T431151#12091964 (10HNordeenWMF) [20:10:59] 06Machine-Learning-Team, 06App Experience: Articles on men are appearing as suggestions to users based on a stated interested in women biographies - https://phabricator.wikimedia.org/T431151#12092011 (10HNordeenWMF) {F92129361} The Growth team also shows this topic within Suggested Edits, and so experiences t... [20:15:10] 06Machine-Learning-Team, 06App Experience: Articles on men are appearing as suggestions to users based on a stated interested in women biographies - https://phabricator.wikimedia.org/T431151#12092050 (10Dbrant) While it is true that the corresponding Wikidata item might have more accurate gender information ab... [20:39:15] 06Machine-Learning-Team, 06App Experience: Articles on men are appearing as suggestions to users based on a stated interested in women biographies - https://phabricator.wikimedia.org/T431151#12092146 (10TLessa-WMF) More examples of this issue here. {F92131706} {F92131705} [21:13:38] 06Machine-Learning-Team, 10ORES, 10MediaWiki-Recent-changes, 06Moderator-Tools-Team: ORES flagging not working on simplewiki - https://phabricator.wikimedia.org/T431159#12092266 (10Aklapper) p:05Unbreak!→03Triage @LeastAction: Please see https://www.mediawiki.org/wiki/Phabricator/Project_management#Set... [21:54:06] 06Machine-Learning-Team, 06App Experience: Articles on men are appearing as suggestions to users based on a stated interested in women biographies - https://phabricator.wikimedia.org/T431151#12092348 (10KStoller-WMF) >>! In T431151#12092009, @HNordeenWMF wrote: > {F92129361} > The Growth team also shows this... [22:50:16] 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 06ServiceOps new, 10ServiceOps-SharedInfra, 13Patch-For-Review: Investigate X-Wikimedia-Debug routing for api.wikimedia.org - https://phabricator.wikimedia.org/T428909#12092477 (10Scott_French) Nice analysis in T428909#12070270 @Clement_Goubert. +1 to... [23:40:12] 06Machine-Learning-Team, 06Data-Persistence, 06DBA, 06ServiceOps new, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): codfw: rack B6 maintenance - https://phabricator.wikimedia.org/T430922#12092631 (10Eevans) [23:42:18] 06Machine-Learning-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12092634 (10Eevans)