[09:25:22] morning! [09:30:41] Hey folks, good morning! [10:24:45] \o [10:28:15] 10Machine-Learning-Team: Implement batch prediction for revertrisk-multilingual - https://phabricator.wikimedia.org/T355656 (10achou) [10:28:41] 10Machine-Learning-Team: Implement batch prediction for revertrisk-multilingual - https://phabricator.wikimedia.org/T355656 (10achou) [10:28:43] 10Machine-Learning-Team, 10Goal: Goal: Lift Wing users can request multiple predictions using a single request. - https://phabricator.wikimedia.org/T348153 (10achou) [10:32:32] hey Tobias! [10:44:13] ping me if-when you get a chance to test attaching to exp pods in staging. If it works, I can push the admin-ng stuff to prod as well [10:47:38] I'm on it now [10:51:26] I got failures both when trying to delete a pod and also on exec [10:51:39] I ran the following commands: [10:51:39] `kubectl delete pod revertrisk-wikidata-predictor-default-00013-deployment-6686jhgk` [10:51:49] `kubectl exec -it revertrisk-wikidata-predictor-default-00013-deployment-6686jhgk -c kserve-container -- /bin/bash` [10:52:17] what kind of error? [10:52:30] `Error from server (Forbidden): pods "revertrisk-wikidata-predictor-default-00013-deployment-6686jhgk" is forbidden: User "experimental" cannot delete resource "pods" in API group "" in the namespace "experimental"` [10:52:47] hmmm [10:53:06] I'll have a look after lunch [10:54:11] ack [10:55:23] thanks! it seems to me that we should be using experimental-debug as a user. Otherwise we need to set `experimental` as a user in the rbac configuration. [10:56:02] can the user be specified on the kybectl cmdline? [10:57:17] I have no idea. I use kube_env. e.g. `kube_env experimental ml-staging-codfw` . there is no option for experimental-debug though [10:58:01] no kubectl, not env [11:00:12] I can do `kubectl get pods --user=experimental-debug` but it results in [11:00:12] `error: auth info "experimental-debug" does not exist` [11:00:54] mh, yeah [11:01:13] I'll take a look in a bit and update you on what I find [11:01:39] sure, thanks! [11:15:59] SO the k8s side works, but it's a bit of a kube_env mess [11:17:26] basically, you have to run kube-env normally, and then `export KUBECONFIG=/etc/kubernetes/experimental-debug-ml-staging-codfw.config` [11:25:38] greeat, it works! [11:26:52] I'll have a chat with the k8s guys if we can make this simpler/easier, but at least you're unblocked for now [12:05:22] yeah , super thanks! [12:15:26] One upside of the extra step: it's harder to accidentally knock things over :) [12:47:36] * isaranto lunch! [13:48:00] 10Machine-Learning-Team, 10Goal: Goal: Inference Optimization for Hugging face/Pytorch models - https://phabricator.wikimedia.org/T353337 (10isarantopoulos) We now have access to be able to do operations on running pods in ml-staging-codfw edit/exec/delete) so we can start working directly on the GPU. [14:04:32] isaranto: I now also pushed the exp NS change to prod [14:04:46] 🎉 [17:00:19] 10Machine-Learning-Team, 10SRE, 10Patch-For-Review: Requesting write access to ml-staging-codfw for ML team - https://phabricator.wikimedia.org/T354516 (10klausman) This has been solved for now, though needs better docs and possibly simplification, as an extra step is needed: `lang=shell $ kube_env experime... [17:03:28] yesssss [17:12:14] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10ORES, 10MW-1.41-notes (1.41.0-wmf.26; 2023-09-12), 10MW-1.42-notes (1.42.0-wmf.15; 2024-01-23): ORES Extension master branch is failing tests - https://phabricator.wikimedia.org/T345922 (10Samwalton9-WMF) a:05jsn.sherman→03None [17:35:56] logging off today o/ [17:45:31] good night aiko ! [18:28:31] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Growth-Team, 10Wikipedia-Android-App-Backlog, and 2 others: Add revertrisk-language-agnostic to RecentChanges filters - https://phabricator.wikimedia.org/T348298 (10Samwalton9-WMF) [18:48:53] I updated the patch for locust, will write a readme and open it up for reviews in the morning [18:49:00] logging off folks, cu tomorrow! [18:51:20] night isaranto! [20:05:14] (03PS1) 10Kosta Harlan: revertrisk: Fix i18n message [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992492 (https://phabricator.wikimedia.org/T348298) [20:06:34] (03PS1) 10Kosta Harlan: revertrisk: Fix i18n message reference [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992493 (https://phabricator.wikimedia.org/T348298) [20:32:48] (03CR) 10Jsn.sherman: [C: 04-1] "Thanks for your work on this! I didn't think to check the "what's this?" dialog before. I do notice that the url in the ores-rcfilters-rev" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992493 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [20:34:47] (03CR) 10Jsn.sherman: [C: 03+2] "Actually, forget that feedback for this patch, it can be addressed elsewhere. Thanks for the fix!" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992493 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [20:35:44] (03CR) 10Jsn.sherman: [C: 03+1] "I noticed that the url in the ores-rcfilters-revertrisklanguageagnostic-whats-this-body message key is not getting rendered into a link, s" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992492 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [20:38:02] (03Merged) 10jenkins-bot: revertrisk: Fix i18n message reference [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992493 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [20:41:26] (03PS2) 10Kosta Harlan: revertrisk: Fix i18n messages [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992492 (https://phabricator.wikimedia.org/T348298) [20:41:43] (03PS1) 10Kosta Harlan: revertrisk: Fix i18n message reference [extensions/ORES] (wmf/1.42.0-wmf.15) - 10https://gerrit.wikimedia.org/r/992506 (https://phabricator.wikimedia.org/T348298) [20:43:52] (03CR) 10Kosta Harlan: "Done, thanks." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992492 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [20:45:46] (03CR) 10Jsn.sherman: [C: 03+2] "looks good; thanks!" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992492 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [20:48:35] (03Merged) 10jenkins-bot: revertrisk: Fix i18n messages [extensions/ORES] - 10https://gerrit.wikimedia.org/r/992492 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [20:48:55] (03PS1) 10Kosta Harlan: revertrisk: Fix i18n messages [extensions/ORES] (wmf/1.42.0-wmf.15) - 10https://gerrit.wikimedia.org/r/992507 (https://phabricator.wikimedia.org/T348298) [21:01:50] (03CR) 10TrainBranchBot: [C: 03+2] "Approved by kharlan@deploy2002 using scap backport" [extensions/ORES] (wmf/1.42.0-wmf.15) - 10https://gerrit.wikimedia.org/r/992506 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [21:01:58] (03CR) 10TrainBranchBot: [C: 03+2] "Approved by kharlan@deploy2002 using scap backport" [extensions/ORES] (wmf/1.42.0-wmf.15) - 10https://gerrit.wikimedia.org/r/992507 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [21:04:39] (03Merged) 10jenkins-bot: revertrisk: Fix i18n message reference [extensions/ORES] (wmf/1.42.0-wmf.15) - 10https://gerrit.wikimedia.org/r/992506 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan) [21:04:47] (03Merged) 10jenkins-bot: revertrisk: Fix i18n messages [extensions/ORES] (wmf/1.42.0-wmf.15) - 10https://gerrit.wikimedia.org/r/992507 (https://phabricator.wikimedia.org/T348298) (owner: 10Kosta Harlan)