[04:30:48] <wikibugs>	 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Create new mediawiki links change streams based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399#12210537 (10tchin) Coming back to this after talking with the people at the Internet Archive about improving page...
[07:22:01] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-07 - 2026-08-28): LLM observability - https://phabricator.wikimedia.org/T429597#12210734 (10Gehel)
[07:23:06] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-07 - 2026-08-28): monitoring: Grafana dashboard for LLM serving on MI300X - https://phabricator.wikimedia.org/T429237#12210736 (10Gehel)
[07:23:33] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-07 - 2026-08-28): monitoring: View GPU usage per LLM deployment/model - https://phabricator.wikimedia.org/T429236#12210744 (10Gehel)
[07:24:16] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-07 - 2026-08-28): ml-lab: Update pytorch-rocm to a current ROCm-compatible release - https://phabricator.wikimedia.org/T428023#12210757 (10Gehel)
[07:35:06] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12210790 (10Gehel)
[07:35:22] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 06DC-Ops, 10ops-eqiad, 06SRE, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12210792 (10Gehel) @RobH : this seems to be completed on our side. I'll keep t...
[07:35:52] <wikibugs>	 06Machine-Learning-Team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 07Essential-Work: Document procedure for updating the underlying distro for model servers (e.g. Bullseye -> Bookworm) - https://phabricator.wikimedia.org/T352189#12210796 (10Gehel)
[07:38:34] <wikibugs>	 (03CR) 10Bartosz Wójtowicz: "Looks great ❤️ Adding 2 small comments" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1324755 (https://phabricator.wikimedia.org/T433569) (owner: 10Kevin Bazira)
[07:54:42] <wikibugs>	 (03PS1) 10Bartosz Wójtowicz: revise-tone-task-generator: Remove topic filtering. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1325289 (https://phabricator.wikimedia.org/T433319)
[08:13:46] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27): Create dataset of article titles for TTS V1 - https://phabricator.wikimedia.org/T432128#12210927 (10BWojtowicz-WMF) The second version of the dataset is now created and available here: {F98314219}  The improvements described above fixed a lot of the issues we experienced w...
[08:59:57] <wikibugs>	 (03CR) 10AikoChou: [C:03+1] "LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1325289 (https://phabricator.wikimedia.org/T433319) (owner: 10Bartosz Wójtowicz)
[09:02:22] <wikibugs>	 (03PS3) 10Kevin Bazira: qwen36: return the thinking trace separately from the answer [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1324755 (https://phabricator.wikimedia.org/T433569)
[09:03:57] <wikibugs>	 (03CR) 10Bartosz Wójtowicz: [C:03+2] revise-tone-task-generator: Remove topic filtering. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1325289 (https://phabricator.wikimedia.org/T433319) (owner: 10Bartosz Wójtowicz)
[09:05:29] <wikibugs>	 (03Merged) 10jenkins-bot: revise-tone-task-generator: Remove topic filtering. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1325289 (https://phabricator.wikimedia.org/T433319) (owner: 10Bartosz Wójtowicz)
[09:24:29] <wikibugs>	 (03CR) 10Kevin Bazira: qwen36: return the thinking trace separately from the answer (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1324755 (https://phabricator.wikimedia.org/T433569) (owner: 10Kevin Bazira)
[09:47:23] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Resolve issues with python-mwapi dependency on LiftWing - https://phabricator.wikimedia.org/T434763 (10BWojtowicz-WMF) 03NEW
[09:48:56] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 07Essential-Work: Introduce re-try mechanisms for MW API requests in LiftWing models - https://phabricator.wikimedia.org/T407843#12211362 (10BWojtowicz-WMF) This task should be picked up only after we resolve the work tracked in here https://phabricator.wikimedia.org/T43...
[09:55:10] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27): Update Revise Tone pipelines to incorporate additional languages and update topic filters - https://phabricator.wikimedia.org/T433319#12211377 (10BWojtowicz-WMF) We have now disabled the topic filtering alltogether.   The latency seems to have neither increased nor decreas...
[10:01:06] <wikibugs>	 (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you for fixing the comments, all looks great now! ❤️" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1324755 (https://phabricator.wikimedia.org/T433569) (owner: 10Kevin Bazira)
[10:15:19] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Pods evicted on ml-serve-eqiad due to disk pressure - https://phabricator.wikimedia.org/T431017#12211434 (10BWojtowicz-WMF) FYI we have added `ephemeral-storage` requests/limits to all predictor containers that we are servin...
[10:27:23] <wikibugs>	 (03CR) 10Kevin Bazira: [C:03+2] qwen36: return the thinking trace separately from the answer [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1324755 (https://phabricator.wikimedia.org/T433569) (owner: 10Kevin Bazira)
[10:30:08] <wikibugs>	 (03Merged) 10jenkins-bot: qwen36: return the thinking trace separately from the answer [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1324755 (https://phabricator.wikimedia.org/T433569) (owner: 10Kevin Bazira)
[11:10:44] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 10Wikidata, 13Patch-For-Review, 06Wikidata-Omega (Triaging): Threshold analysis for WMDE use cases of Wikidata RR model - https://phabricator.wikimedia.org/T429049#12211615 (10gkyziridis) You're right on this base rate, the recall-based bins add a lot of false positi...
[11:27:09] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27): [draft] RR-Wikidata: Decide on Linked Artifact Cache for stream (WME consumption via Hoarde) - https://phabricator.wikimedia.org/T433699#12211658 (10gkyziridis) **Update: internal discussion**  The ML team discussed this task in a recent team meeting.  The discussion centr...
[11:44:59] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27), 10Wikidata, 13Patch-For-Review, 06Wikidata-Omega (Triaging): Threshold analysis for WMDE use cases of Wikidata RR model - https://phabricator.wikimedia.org/T429049#12211774 (10achou) Thanks for working on this, George!  A few comments and questions: * Ground truth Re...
[12:42:45] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27): qwen36: Support thinking mode on the OpenAI-compatible endpoints - https://phabricator.wikimedia.org/T433569#12212019 (10kevinbazira) We can now turn on thinking for each request on the chat completions endpoint by sending `"chat_template_kwargs":{"enable_thinking":true}`....
[12:43:15] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27): qwen36: Support thinking mode on the OpenAI-compatible endpoints - https://phabricator.wikimedia.org/T433569#12212021 (10kevinbazira)
[13:14:04] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27): qwen36: Support thinking mode on the OpenAI-compatible endpoints - https://phabricator.wikimedia.org/T433569#12212166 (10kevinbazira) The isvc now returns the thinking trace in its own `reasoning` field. The trace is not in the answer text. This applies to both non-streame...
[13:14:36] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27): qwen36: Support thinking mode on the OpenAI-compatible endpoints - https://phabricator.wikimedia.org/T433569#12212168 (10kevinbazira)
[14:39:36] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27): qwen36: Support thinking mode on the OpenAI-compatible endpoints - https://phabricator.wikimedia.org/T433569#12212610 (10kevinbazira)
[15:05:22] <wikibugs>	 06Machine-Learning-Team, 06Data-Engineering, 10Event-Platform: Create new mediawiki links change streams based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399#12212767 (10Ottomata) > Does this mean having a sort of PageRenderingGenerated domain event should be a prerequisi...
[19:39:58] <wikibugs>	 06Machine-Learning-Team (Q1 FY2026-27): [draft] RR-Wikidata: Decide on Linked Artifact Cache for stream (WME consumption via Hoarde) - https://phabricator.wikimedia.org/T433699#12214010 (10Ottomata) Nice! I think for LAC uses that need to be triggered by edits, we can you all off of change prop entirely.  The bu...