[00:00:51] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10RZamora-WMF)
[07:16:00] <isaranto>	 Guten Morgen Alle!
[07:28:05] <elukey>	 o/
[08:12:49] <Guest5405>	 (This is aiko) o/ morning
[08:13:01] <isaranto>	 hello Guest5405 !!
[08:13:05] <isaranto>	 :)
[08:14:25] <elukey>	 Guest5405: what happened to your nick? :)
[08:29:19] <aiko>	 hello
[08:33:05] <elukey>	 hello aiko!
[08:34:07] <aiko>	 I don't know what happened but my nick is back :)
[09:06:29] <isaranto>	 elukey: o/
[09:06:33] <isaranto>	 we're in the meeting
[09:06:37] <elukey>	 ouch
[09:06:47] <elukey>	 sorry I got distracted by puppet and other things :D
[09:14:41] <wikibugs>	 10Machine-Learning-Team: Add User-agent in header of Ores extension - https://phabricator.wikimedia.org/T342605 (10isarantopoulos)
[09:18:43] <wikibugs>	 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: Add User-agent in header of Ores extension - https://phabricator.wikimedia.org/T342605 (10isarantopoulos)
[09:41:20] <isaranto>	 klausman: sry I forgot to mention I can do the user agent change (if anything is required), since I have been working with the extension's php code 
[09:42:01] <klausman>	 sgtm, it probably takes you 5m wheere it would take me 30m
[10:26:55] <wikibugs>	 10Machine-Learning-Team: Revert Risk multi-lingual model performance and reliability may need a review - https://phabricator.wikimedia.org/T340822 (10klausman) I've also run the same test tool against RR-ML and got much worse latency overall (though no errors, which is great):  `lang=plain $ go run . -p 20 -d 60...
[10:37:36] * elukey lunch!
[10:39:46] <klausman>	 ditto
[11:01:26] <chrisalbon_>	 Morning all
[11:05:53] <klausman>	 heyo Chris
[11:05:58] <aiko>	 morning Chris
[11:09:19] <aiko>	 I sent an email to Philadelphia WikiSalon asking about ORES UI and batched requests
[11:35:35] <isaranto>	 o/ Chris!
[11:35:44] <isaranto>	 nice aiko:!
[11:35:57] * isaranto goes for lunch
[12:16:34] <elukey>	 aiko: thanks!
[12:37:55] <klausman>	 chrisalbon_: https://docs.google.com/document/d/1VbbXQfIU7hKhYhJ2w3dQx8QyZeqL4GVElbnFNxQrmwI/edit?disco=AAAA1fuOLzo Added a list of things we maybe should change/update regarding deprecation of ORES or otherwise follow-up.
[13:06:13] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: use wikiID in inference name on LW for revscoring models - https://phabricator.wikimedia.org/T342266 (10isarantopoulos) In order to roll these changes out and not have any service interruption, we will first deploy the changes for the inference services. We have add...
[13:07:43] <isaranto>	 I wrote what we discussed in the task above. We can proceed with deployments in this patch https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/939744
[13:08:11] <isaranto>	 and then the api gateway changes once we make sure there is no traffic on the pods
[13:08:52] <klausman>	 :+1:
[13:09:35] <isaranto>	 elukey: I need some clarification for a comment on the patch where you mentioned sth about transition rules in api gateway. It sounds great but I havent found anything relevant
[13:12:12] <elukey>	 isaranto: I reviewed the change, why do we need to add "wiki" to all the isvs? I thought we only wanted to do it for the corner case ones
[13:12:41] <elukey>	 ah no ok it was there before, okok
[13:12:45] <elukey>	 sorry pebcack
[13:12:53] <elukey>	 *pebcak, too many changes flying
[13:13:18] <elukey>	 re: api-gateway - with your solution everything seems good, namely we'll keep supporting the corner cases
[13:13:19] <isaranto>	 we change the variable .wiki to reflect wikiID (e.g. 'enwiki') instead of prefix (e.g. 'en')
[13:13:21] <elukey>	 so I am happy with it
[13:13:34] <elukey>	 yes yes I thought something different
[13:13:36] <elukey>	 all good
[13:13:46] <isaranto>	 so we don't deploy api-gateway yet, otherwise we would drop support for now
[13:13:55] <elukey>	 exactly yes, +1
[13:14:04] <elukey>	 in this way we can do it safely at the end
[13:14:43] <isaranto>	 ack, onboard 100%
[13:37:27] <wikibugs>	 10Machine-Learning-Team, 10Documentation: Documentation to migrate an ORES request to LW - https://phabricator.wikimedia.org/T342523 (10isarantopoulos) I started adding some documentation in the ORES Wikitech page https://wikitech.wikimedia.org/wiki/ORES under the section `Guide to migrate from ORES to Lift Wing`
[13:44:23] <klausman>	 isaranto: something weird is going on with that table. Want me to investigate a fix?
[13:50:24] <isaranto>	 klausman: you mean with the table in the documentation?
[13:50:26] <elukey>	 ok thanks to Filippo I think I know why we don't see all access logs in the dashboard
[13:50:29] <elukey>	 https://phabricator.wikimedia.org/T313099
[13:50:34] <klausman>	 isaranto:yeah
[13:50:44] <elukey>	 there is a rule to drop 99% of logs from containers called istio-proxy
[13:50:46] <isaranto>	 if it is easy for you please yes!
[13:50:47] * elukey cries in a corner
[13:51:46] <isaranto>	 elukey: is this the reason u mentioned why we dont see any request coming from curl?
[13:51:57] <elukey>	 I believe so yes
[14:08:51] <klausman>	 isaranto: fixed it. But boy do I dislike Wiki markup even more now.
[14:29:54] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Fix model row upsert warning [extensions/ORES] - 10https://gerrit.wikimedia.org/r/939240 (https://phabricator.wikimedia.org/T319170) (owner: 10Ladsgroup)
[14:35:52] <wikibugs>	 10Machine-Learning-Team: Add deprecation message for ORES UI - https://phabricator.wikimedia.org/T342118 (10isarantopoulos)
[14:36:22] <isaranto>	 I changed the task above  from creation of UI to adding deprecation message
[14:39:36] <wikibugs>	 10Machine-Learning-Team: Add deprecation message for ORES UI - https://phabricator.wikimedia.org/T342118 (10kevinbazira)
[14:40:26] <kevinbazira>	 Thanks isaranto. I fixed something small in the description :)
[14:43:44] <isaranto>	 thanks!
[14:46:26] <wikibugs>	 (03Merged) 10jenkins-bot: Fix model row upsert warning [extensions/ORES] - 10https://gerrit.wikimedia.org/r/939240 (https://phabricator.wikimedia.org/T319170) (owner: 10Ladsgroup)
[14:55:51] <wikibugs>	 10Machine-Learning-Team: Add deprecation messages for features not supported in ores-legacy - https://phabricator.wikimedia.org/T342663 (10isarantopoulos)
[14:56:30] <wikibugs>	 10Machine-Learning-Team: Add deprecation messages for features not supported in ores-legacy - https://phabricator.wikimedia.org/T342663 (10isarantopoulos)
[14:56:59] <isaranto>	 starting deployments for goodfaith, damaging and reverted in ml-staging
[14:57:09] <elukey>	 +1
[15:09:48] <isaranto>	 starting eqiad!
[15:10:53] <isaranto>	 I'll write here if I have the same issues with the revisions as last time (since we are re-deploying all pods with new chart version)
[15:19:07] <isaranto>	 klausman: thanks for fixing that table!
[15:21:17] <wikibugs>	 10Machine-Learning-Team: Add deprecation messages for features not supported in ores-legacy - https://phabricator.wikimedia.org/T342663 (10isarantopoulos) The error message should include a 400 error - Bad response.
[15:32:08] <elukey>	 we have complete access logs!
[15:32:40] <elukey>	 dashboard is https://logstash.wikimedia.org/app/dashboards#/view/138271f0-40ce-11ed-bb3e-0bc9ce387d88?_g=h@c823129&_a=h@dc7e983
[15:33:11] <elukey>	 (remember to filter for ml-ctrl-something in the first top-left graph, we share logs with serviceops)
[15:37:35] <klausman>	 very nice.
[15:43:22] <isaranto>	 splendid! :tada
[15:43:28] <isaranto>	 🎉
[15:44:19] <isaranto>	 all olds pods terminate as expected in codfw and eqiad
[15:46:36] <aiko>	 wow nice!!
[15:56:04] <kevinbazira>	 🎉🎉🎉
[16:04:24] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: use wikiID in inference name on LW for revscoring models - https://phabricator.wikimedia.org/T342266 (10isarantopoulos) The changes for the new inference services have been deployed in codfw and eqiad and the httpb tests have been update to reflect the current statu...
[16:06:09] <isaranto>	 game on!
[16:06:19] <isaranto>	 Amir1: o/
[16:06:41] <isaranto>	 after a bunch of changes we are ready to deploy eswikibooks and eswikiquote! https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/939697
[16:07:44] <elukey>	 isaranto: do we need to deploy the api-gateway changes too first?
[16:08:08] <isaranto>	 no since the extension uses internal endpoints
[16:08:18] <elukey>	 right right only for the extension, super
[16:08:23] <elukey>	 double checking :)
[16:08:59] <isaranto>	 :) and you're right as these things can easily falls through the cracks
[16:53:19] <klausman>	 \o ehading out. have a nice evening, folks.
[16:53:56] <elukey>	 me too, have a nice rest of the day!
[17:16:02] <Amir1>	 isaranto: awesome, for when do you want it deployed?
[17:38:23] <isaranto>	 Amir1: we can do it tomorrow. It would be great if we could do it together so I know what to check etc from mediawiki side,then the rest of the deployments will be smoother(famous last words:). Let me know what works best for you.thanks!
[17:38:50] <Amir1>	 sure, ping me tomorrow and let's do it
[20:51:04] <wikibugs>	 10Machine-Learning-Team, 10Research: Index out of range in revert risk multi-lingual - https://phabricator.wikimedia.org/T340811 (10Trokhymovych) I have checked the proposed changes (MR: https://gitlab.wikimedia.org/repos/research/knowledge_integrity/-/merge_requests/17).   As for that, I used the random sampl...
[21:43:26] <wikibugs>	 10Machine-Learning-Team, 10Research: Index out of range in revert risk multi-lingual - https://phabricator.wikimedia.org/T340811 (10Isaac) great news @Trokhymovych and thanks for sharing this results table! if there are any specific revids that were erroring out, don't hesitate to let me know which plus what t...