[08:01:48] 10Machine-Learning-Team, 10SRE, 10serviceops: Import and deploy istio 1.15.7 - https://phabricator.wikimedia.org/T334068 (10elukey) [08:02:47] 10Machine-Learning-Team, 10SRE, 10serviceops: Import and deploy istio 1.15.7 - https://phabricator.wikimedia.org/T334068 (10elukey) Rollout to ml-serve/aux/dse completed. To keep archives happy, I used ` istioctl-1.15.7 upgrade -f config.yaml` Last step: rollout to wikikube clusters [08:03:04] 10Machine-Learning-Team, 10SRE, 10serviceops: Import and deploy istio 1.15.7 - https://phabricator.wikimedia.org/T334068 (10elukey) a:05elukey→03JMeybohm [08:27:10] isaranto: chart merged, great work :) [08:28:38] Nice, thanks for the help! [08:42:04] when we have the pods deployed I'll create a task to add the LVS endpoint to staging [08:42:21] so that we'll be able to curl something like ores-legacy.svc.codfw.wmnet [08:42:35] err ores-legacy-staging.svc.codfw.wmnet [08:45:49] Cool! I opened the other patch with the helmfile and the values for staging [08:47:05] Pups,CI failed. Lemme check and I ll send a review request [08:47:13] *oups [08:47:25] A yeah of course I have to tebase now that we merged [08:47:56] 10Machine-Learning-Team, 10ORES, 10Wikimedia Enterprise: Investigate tools that use ORES - https://phabricator.wikimedia.org/T330854 (10elukey) @prabhat thanks a lot for the follow up! Yes the performance difference is visible for the MVP state of Lift Wing, we decided (for the moment) to avoid the complex c... [08:48:12] https://phabricator.wikimedia.org/T330854#8773369 is very interesting [08:48:25] I am wondering if we'd need the score-cache or not at the end [08:59:26] 10Machine-Learning-Team, 10SRE, 10serviceops: Import and deploy istio 1.15.7 - https://phabricator.wikimedia.org/T334068 (10JMeybohm) 05Open→03Resolved Thanks! Wikikube is done as well [08:59:36] 10Machine-Learning-Team, 10SRE, 10serviceops: Import and deploy istio 1.15.7 - https://phabricator.wikimedia.org/T334068 (10JMeybohm) [09:08:55] We'll need to see some usage patterns to take a decision on that, cause it also has to do with the use case [09:10:02] for the revscoring models input is more standard ( a revid) so if we monitor the usage and see that some revisions are requested multiple times it would make sense [09:30:02] ok, the patch is ready now https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/908191 [09:30:59] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 15th round of wikis - https://phabricator.wikimedia.org/T308141 (10kevinbazira) Model evaluation has been completed and below are the backtesting results: | | Precision@0.5 | Recall@0.5 |quwiki | 0.75 | 0.16 |rmwiki |... [09:36:03] I just checked and on dse-k8s-worker1001 and 1002 we have two GPUs each [09:41:23] 0.0 [09:53:35] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-notice: Deploy "add a link" to 8th round of wikis - https://phabricator.wikimedia.org/T308133 (10Sgs) a:05Sgs→03None >>! In T308133#8759296, @Trizek-WMF wrote: > I tested the models, and they are all worki... [09:55:43] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 9th round of wikis - https://phabricator.wikimedia.org/T308134 (10Sgs) >>! In T308134#8759334, @Trizek-WMF wrote: > I tested the models, and everything works fine, except: > * **[[https://hr.wikipedia... [10:35:51] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10Marostegui) @ayounsi we are placing new DB hosts in production, can you run the same query you ran to gather the affected DBs just in c... [10:41:31] * elukey lunch! [12:17:45] 10Machine-Learning-Team, 10Spike: [Spike] Run models and frameworks on AMD GPU and identify challenges - https://phabricator.wikimedia.org/T334583 (10isarantopoulos) [12:31:18] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ayounsi) [12:31:48] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10ayounsi) [12:32:02] 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ayounsi) [12:34:45] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ayounsi) [12:35:38] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10ayounsi) >>! In T333377#8775126, @Marostegui wrote: > @ayounsi we are placing new DB hosts in production, can you run the same query yo... [12:38:24] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row D switches upgrade - https://phabricator.wikimedia.org/T333377 (10Marostegui) Thank you, nothing changes from our DB side! [13:00:21] (03PS2) 10AikoChou: WIP - events: add code to generate predicted_classification events [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/907923 (https://phabricator.wikimedia.org/T331401) [13:02:03] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10Ottomata) [13:02:18] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10Ottomata) When we do this, we should also consider all the other kinds of mediawiki 'links'.... [13:59:13] 10artificial-intelligence: Use AI to automatically generate edit summaries - https://phabricator.wikimedia.org/T334598 (10Bugreporter2) [14:01:18] 10Machine-Learning-Team, 10artificial-intelligence: Use AI to automatically generate edit summaries - https://phabricator.wikimedia.org/T334598 (10Bugreporter2) [14:08:39] 10Machine-Learning-Team, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 16 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10lbowmaker) [14:28:41] 10artificial-intelligence: Use AI to automatically generate edit summaries - https://phabricator.wikimedia.org/T334598 (10Aklapper) @Bugreporter2: It is up to teams what teams plan to have on their workboards, thus removing tag [14:42:29] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10User-notice: Deploy "add a link" to 8th round of wikis - https://phabricator.wikimedia.org/T308133 (10Sgs) a:03Sgs [15:06:18] 10Machine-Learning-Team, 10Analytics-Radar, 10Data-Engineering-Icebox, 10Patch-For-Review: Upgrade ROCm to 4.5 - https://phabricator.wikimedia.org/T295661 (10elukey) First attempt on dse-k8s-worker1001 ended up in some errors, among them: ` The following packages have unmet dependencies: rocm-llvm : Dep... [15:06:55] sigh --^ [15:23:40] 10Machine-Learning-Team, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 16 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10fgiunchedi) [15:30:50] 10Lift-Wing, 10Machine-Learning-Team, 10Research (FY2022-23-Research-January-March): Create a language agnostic model to predict reverts on Wikipedia - https://phabricator.wikimedia.org/T314385 (10diego) 05In progress→03Resolved [15:34:40] 10Lift-Wing, 10Machine-Learning-Team, 10Research (FY2022-23-Research-January-March): Create a language agnostic model to predict reverts on Wikipedia - https://phabricator.wikimedia.org/T314385 (10diego) [15:42:12] 10Machine-Learning-Team, 10Analytics-Radar, 10Data-Engineering-Icebox, 10Patch-For-Review: Upgrade ROCm to 4.5 - https://phabricator.wikimedia.org/T295661 (10elukey) Had a chat with Moritz, and here some relevant readings: * https://github.com/RadeonOpenCompute/ROCm/issues/1125 * https://github.com/ROCm-D... [15:46:09] 10Machine-Learning-Team, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 16 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Michaelcochez) Do we have a timeline for this move already? Is it better to not update the gerrit repo at the moment? Ou... [15:46:38] going afk folks! Have a nice rest of the day :) [15:52:06] 10Machine-Learning-Team, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 16 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10calbon) [15:52:50] 10Machine-Learning-Team, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 16 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10leila) [15:53:34] 10Machine-Learning-Team: Move add-a-link to gitlab - https://phabricator.wikimedia.org/T334605 (10calbon) [15:55:56] have a great evening all! [16:33:27] I'm logging off as well folks, have a great rest of day, evening :) [17:15:09] 10Machine-Learning-Team, 10GitLab (Project Migration): Move add-a-link to gitlab - https://phabricator.wikimedia.org/T334605 (10Aklapper) [17:18:50] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-notice: Deploy "add a link" to 7th round of wikis - https://phabricator.wikimedia.org/T304551 (10Sgs) [17:19:46] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-notice: Deploy "add a link" to 7th round of wikis - https://phabricator.wikimedia.org/T304551 (10Sgs) 05In progress→03Open [19:20:29] 10Machine-Learning-Team, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 16 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10thcipriani) [19:24:09] 10Machine-Learning-Team, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 16 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10thcipriani) [19:26:27] 10Machine-Learning-Team, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 16 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10thcipriani) [19:50:28] 10Machine-Learning-Team, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 16 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10bd808)