[05:49:35] (03CR) 10Elukey: "Do you think that there is a way to get the logging level from an environment variable? So we'd be able to turn it off/on without rebuildi" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937506 (owner: 10Ilias Sarantopoulos) [05:52:46] (03PS1) 10Santhosh: langid: Provide wiki language code also in outputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) [05:52:54] (03CR) 10CI reject: [V: 04-1] langid: Provide wiki language code also in outputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) (owner: 10Santhosh) [05:53:26] 10Lift-Wing, 10Machine-Learning-Team, 10I18n, 10NewFunctionality-Worktype, 10Patch-For-Review: Create a language detection service in LiftWing - https://phabricator.wikimedia.org/T340507 (10santhosh) p:05Triage→03Medium [06:12:09] (03PS2) 10Santhosh: langid: Provide wiki language code and name also in outputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) [06:12:16] (03CR) 10CI reject: [V: 04-1] langid: Provide wiki language code and name also in outputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) (owner: 10Santhosh) [06:18:56] (03PS3) 10Santhosh: langid: Provide wiki language code and name also in outputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) [06:22:30] (03CR) 10CI reject: [V: 04-1] langid: Provide wiki language code and name also in outputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) (owner: 10Santhosh) [06:23:36] (03PS29) 10Kevin Bazira: Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) [06:25:33] (03PS4) 10Santhosh: langid: Provide wiki language code and name also in outputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) [06:49:08] (03CR) 10CI reject: [V: 04-1] Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) (owner: 10Kevin Bazira) [07:07:45] (03PS30) 10Kevin Bazira: Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) [07:32:42] (03CR) 10CI reject: [V: 04-1] Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) (owner: 10Kevin Bazira) [07:41:47] (03CR) 10Ilias Sarantopoulos: ores-legacy: increase logging level to DEBUG (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937506 (owner: 10Ilias Sarantopoulos) [07:53:59] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MW-1.41-notes (1.41.0-wmf.17; 2023-07-11), 10Patch-For-Review: Move backend of ORES MediaWiki extension to Lift Wing - https://phabricator.wikimedia.org/T319170 (10isarantopoulos) I am currently debugging the following [[ https://phabricator.wikimedia... [08:18:21] (03PS31) 10Kevin Bazira: Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) [08:43:38] (03CR) 10CI reject: [V: 04-1] Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) (owner: 10Kevin Bazira) [08:59:59] (03PS32) 10Kevin Bazira: Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) [09:13:45] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MW-1.41-notes (1.41.0-wmf.17; 2023-07-11), 10Patch-For-Review: Move backend of ORES MediaWiki extension to Lift Wing - https://phabricator.wikimedia.org/T319170 (10Ladsgroup) >>! In T319170#9011577, @isarantopoulos wrote: > I am currently debugging th... [09:24:57] (03CR) 10CI reject: [V: 04-1] Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) (owner: 10Kevin Bazira) [09:56:45] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "LGTM! I added a suggestion but feel free to disregard it." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) (owner: 10Santhosh) [10:27:31] * elukey lunch [10:36:09] (03PS5) 10Santhosh: langid: Provide wiki language code and name also in outputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) [10:37:15] (03CR) 10Ilias Sarantopoulos: [C: 03+1] langid: Provide wiki language code and name also in outputs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) (owner: 10Santhosh) [10:39:27] (03CR) 10Santhosh: langid: Provide wiki language code and name also in outputs (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) (owner: 10Santhosh) [10:46:37] (03CR) 10Ilias Sarantopoulos: langid: Provide wiki language code and name also in outputs (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) (owner: 10Santhosh) [11:17:26] (03CR) 10Santhosh: langid: Provide wiki language code and name also in outputs (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) (owner: 10Santhosh) [11:38:59] 10Machine-Learning-Team: Revert Risk multi-lingual model performance and reliability may need a review - https://phabricator.wikimedia.org/T340822 (10klausman) Digging into the logging of the kserve and istio containers a bit, I have found a few things: - kserve does not log anything for the bad requests - isti... [11:44:07] * klausman lunch [12:04:02] (03PS33) 10Kevin Bazira: Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) [12:04:24] (03CR) 10CI reject: [V: 04-1] Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) (owner: 10Kevin Bazira) [12:22:08] (03CR) 10Ilias Sarantopoulos: [C: 03+1] langid: Provide wiki language code and name also in outputs (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937575 (https://phabricator.wikimedia.org/T340507) (owner: 10Santhosh) [12:36:35] 10Machine-Learning-Team, 10Research: (stretch) Deploy multilingual readability model to LiftWing - https://phabricator.wikimedia.org/T334182 (10MGerlach) [13:06:39] (03PS34) 10Kevin Bazira: Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) [13:25:47] 10Machine-Learning-Team, 10Research: (stretch) Deploy multilingual readability model to LiftWing - https://phabricator.wikimedia.org/T334182 (10MGerlach) >>! In T334182#9004000, @elukey wrote: > Hi @MGerlach! We follow https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing#Hosting_stages_for_a_model_ser... [13:31:38] (03CR) 10CI reject: [V: 04-1] Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) (owner: 10Kevin Bazira) [13:49:48] (03PS2) 10Ilias Sarantopoulos: ores-legacy: configure logging level from env var [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937506 [13:51:25] elukey: I added logging level override via env var with some basic validation. I found no way to set an environment variable in the .ini file so I used the dictConfig instead [13:52:53] (03CR) 10Elukey: [C: 03+1] "Really nice!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937506 (owner: 10Ilias Sarantopoulos) [13:52:57] isaranto: really nice! [13:53:04] ty [13:53:08] :D [14:08:26] (03PS1) 10Ilias Sarantopoulos: ores-legacy: add error response for v1 requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937960 (https://phabricator.wikimedia.org/T341486) [14:09:45] 10Machine-Learning-Team, 10Patch-For-Review: [ores-legacy] add message that v1 support for ORES has been dropped - https://phabricator.wikimedia.org/T341486 (10isarantopoulos) I have added the following response to requests that use /v1/scores or /scores path (both are ORES v1). ` "error": { "... [14:35:36] 10Machine-Learning-Team, 10ops-codfw: ManagementSSHDown - https://phabricator.wikimedia.org/T341648 (10RhinosF1) [14:45:24] 10Machine-Learning-Team, 10ops-codfw: ManagementSSHDown - https://phabricator.wikimedia.org/T341648 (10elukey) [14:45:29] 10Machine-Learning-Team, 10DC-Ops: hw troubleshooting: iDrac stuck for ores2003.codfw.wmnet - https://phabricator.wikimedia.org/T341657 (10elukey) [14:47:39] (03CR) 10Elukey: "One nit!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937960 (https://phabricator.wikimedia.org/T341486) (owner: 10Ilias Sarantopoulos) [14:48:12] 10Machine-Learning-Team, 10ContentTranslation, 10Research, 10Epic: Verify if the Python recommendation API can support the use cases of the nodejs one - https://phabricator.wikimedia.org/T340854 (10elukey) a:05elukey→03None [14:49:04] 10Machine-Learning-Team, 10Language-Team, 10Epic: Migrate Content Translation Recommendation API to Lift Wing - https://phabricator.wikimedia.org/T308164 (10elukey) [14:49:23] 10Machine-Learning-Team, 10ContentTranslation, 10Research, 10Epic: Verify if the Python recommendation API can support the use cases of the nodejs one - https://phabricator.wikimedia.org/T340854 (10elukey) 05Open→03Stalled Thanks a lot for the inputs Isaac! I think this task should be progressed by who... [14:55:19] 10Machine-Learning-Team, 10Research: (stretch) Deploy multilingual readability model to LiftWing - https://phabricator.wikimedia.org/T334182 (10elukey) >>! In T334182#9012502, @MGerlach wrote: >>>! In T334182#9004000, @elukey wrote: >> Hi @MGerlach! We follow https://wikitech.wikimedia.org/wiki/Machine_Learnin... [14:56:43] 10Machine-Learning-Team, 10ORES, 10Documentation, 10User-AKlapper: Update docs that ORES will be replaced by Lift Wing - https://phabricator.wikimedia.org/T305963 (10elukey) 05Open→03Resolved We are going to publish more docs and a detailed plan very soon about the ORES migration's timings. For the mom... [14:56:45] 10Machine-Learning-Team, 10Epic: Migrate ORES clients to LiftWing - https://phabricator.wikimedia.org/T312518 (10elukey) [15:01:48] (03PS35) 10Kevin Bazira: Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) [15:17:33] isaranto: ok if I merge https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/937506 ? [15:17:37] so we can test it etc.. [15:26:45] (03CR) 10CI reject: [V: 04-1] Set up production and test images for the recommendation-api migration [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/932810 (https://phabricator.wikimedia.org/T339890) (owner: 10Kevin Bazira) [15:29:57] 10Machine-Learning-Team, 10ops-codfw: ManagementSSHDown - https://phabricator.wikimedia.org/T341648 (10Jhancock.wm) found the actual mgmt port to be loose. had server deposed and inspected. bracket that was holding idrac card has broken. tried reseating but port mgmt port goes down shortly after booting up.... [15:47:05] elukey: re ors2003: are we just going to see if we can do without the mgmt card? Or go the epoxy (and battery) route? [15:51:55] klausman: to be honest we can just decom the node if dcops doesn't find a way to fix [15:52:16] There is that option as well. We likely don't need the spare capacity [15:52:40] elukey: I'm merging and deploying before logging off for the day [15:52:46] (03CR) 10Ilias Sarantopoulos: [C: 03+2] ores-legacy: configure logging level from env var [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937506 (owner: 10Ilias Sarantopoulos) [15:53:27] isaranto: nice thanks! [15:53:38] (03Merged) 10jenkins-bot: ores-legacy: configure logging level from env var [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937506 (owner: 10Ilias Sarantopoulos) [15:53:42] On the ticket, Jenn mentions fixing the bracket with epoxy and replacing the battery. But it might not even be worth that effort [15:54:14] if Jenn makes it then it is fine, I told them on the dcops chan that we can decom in case [15:54:34] alrighty [16:08:12] https://github.com/dennistobar/serobot/issues/5 [16:08:24] another report of revert-risk multi-lingual timeouts [16:08:59] (03PS2) 10Ilias Sarantopoulos: ores-legacy: add error response for v1 requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937960 (https://phabricator.wikimedia.org/T341486) [16:09:22] (03CR) 10Ilias Sarantopoulos: ores-legacy: add error response for v1 requests (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/937960 (https://phabricator.wikimedia.org/T341486) (owner: 10Ilias Sarantopoulos) [16:12:55] 10Machine-Learning-Team: Revert Risk multi-lingual model performance and reliability may need a review - https://phabricator.wikimedia.org/T340822 (10elukey) Not a great feedback from https://github.com/dennistobar/serobot/issues/5#issuecomment-1634497760 A bot owner tried to switch to revertrisk multi-lingual... [16:17:01] going afk for today folks [16:17:04] have a nice rest of the day [16:17:46] DEBUG logging level works fine on ores-legacy [16:17:53] deployed to staging and prod [16:18:00] going afk as well! [16:55:41] 10Machine-Learning-Team: Revert Risk multi-lingual model performance and reliability may need a review - https://phabricator.wikimedia.org/T340822 (10diego) Few things: * We are checking if it is possible to simplify the usage of the //mweditypes// package without significantly impact the models' performance.... [19:35:30] 10Machine-Learning-Team, 10Research: Index out of range in revert risk multi-lingual - https://phabricator.wikimedia.org/T340811 (10Isaac) New version released that should address this: https://pypi.org/project/mwedittypes/2.1.0/ The fix for this (using the `skip_style_tags` option) triggered a slew of other...