[04:28:06] 10Machine-Learning-Team: Add deprecation message for too many revision ids - https://phabricator.wikimedia.org/T342789 (10isarantopoulos) a:03isarantopoulos [04:39:13] 10Machine-Learning-Team: Add deprecation message for too many revision ids - https://phabricator.wikimedia.org/T342789 (10isarantopoulos) [04:50:20] (03PS1) 10Ilias Sarantopoulos: ores-legacy: limit number of lw requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942056 (https://phabricator.wikimedia.org/T342789) [04:52:43] (03PS2) 10Ilias Sarantopoulos: ores-legacy: limit number of lw requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942056 (https://phabricator.wikimedia.org/T342789) [04:58:32] 10Machine-Learning-Team, 10Patch-For-Review: Add deprecation message for too many revision ids - https://phabricator.wikimedia.org/T342789 (10isarantopoulos) I have implemented the above adding a 400 HTTP response code . I thought about adding a 410 (endpoint gone) instead but it wouldn't be accurate as we do... [04:59:04] (03PS3) 10Ilias Sarantopoulos: ores-legacy: limit number of lw requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942056 (https://phabricator.wikimedia.org/T342789) [04:59:43] (03PS4) 10Ilias Sarantopoulos: ores-legacy: limit number of lw requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942056 (https://phabricator.wikimedia.org/T342789) [05:07:15] (03PS5) 10Ilias Sarantopoulos: ores-legacy: limit number of lw requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942056 (https://phabricator.wikimedia.org/T342789) [05:08:49] (03PS6) 10Ilias Sarantopoulos: ores-legacy: limit number of lw requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942056 (https://phabricator.wikimedia.org/T342789) [05:26:52] 10Machine-Learning-Team: Users/UserAgents that use ORES - https://phabricator.wikimedia.org/T342835 (10isarantopoulos) [05:27:19] 10Machine-Learning-Team, 10ORES: Users/UserAgents that use ORES - https://phabricator.wikimedia.org/T342835 (10isarantopoulos) [05:27:56] Good morning! I got an early start today due to the weather [05:59:17] o/ [06:03:49] (03CR) 10Elukey: [C: 03+1] "Awesome thank you!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942056 (https://phabricator.wikimedia.org/T342789) (owner: 10Ilias Sarantopoulos) [06:26:43] isaranto: nice work --^ We can deploy and re-test the timeouts to see how much it improves [06:28:54] (03CR) 10Ilias Sarantopoulos: [C: 03+2] ores-legacy: limit number of lw requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942056 (https://phabricator.wikimedia.org/T342789) (owner: 10Ilias Sarantopoulos) [06:29:46] (03Merged) 10jenkins-bot: ores-legacy: limit number of lw requests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942056 (https://phabricator.wikimedia.org/T342789) (owner: 10Ilias Sarantopoulos) [06:30:04] ack [06:33:28] deploying now! [06:34:40] super [06:35:15] how r we going to test the timeouts? [06:35:25] a nevermind [06:35:39] I asked before thinking πŸ˜› [06:36:36] I have a test ready to go on stat1004 [06:36:41] that I used yesterday [06:39:14] super! [06:39:50] I just deployed and I see that also that the envoy image was updated [06:41:02] should be fine, serviceops worked on it [06:41:04] you are ready to go. I checked the request and it gave me the 400 I was expecting. Please check [06:41:04] https://ores-legacy.wikimedia.org/v3/scores/enwiki?models=articletopic|articlequality|damaging|goodfaith&revids=949447954|949447964|949447961|949447982|949447986|949448019|949448035|949448037|949448042|949448059|949448053|949448061|949448062|949448069|949448088|949448103|949448113|949448124|949448126|949448134|949448130|949448142|949448155|949448170|949448172|949448188|949448190|949448196|949448224|949448236|949448243|949448247|949448 [06:41:04] 245|949448248|949448259|949448250|949448265|949448274|949448278|949448277|949448283|949448304|949448307|949448316|949448325|949448327|949448343|949448363|949448372|949448378 [06:41:11] before funning [06:47:36] isaranto: all works! I still see timeouts at first try, so maybe a tweak of the tlsproxy's resources is needed [06:47:44] if not we'll decrease the threshold :) [06:48:03] \o/ [06:50:54] 10Machine-Learning-Team, 10ORES: Users/UserAgents that use ORES - https://phabricator.wikimedia.org/T342835 (10isarantopoulos) a:03isarantopoulos [07:23:00] Afk for an errand for approx 30' [07:30:58] (03PS1) 10AikoChou: ores-legacy: add features field to response models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) [07:31:53] (03PS2) 10AikoChou: ores-legacy: add features field to response models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) [07:34:27] (03CR) 10Elukey: [C: 03+1] Update the ores submodule to deploy deprecation message changes [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/941909 (https://phabricator.wikimedia.org/T342118) (owner: 10Kevin Bazira) [07:41:39] 10Machine-Learning-Team, 10Patch-For-Review: [ores-legacy] Inconsistency when returning features - https://phabricator.wikimedia.org/T342791 (10achou) As per the discussion on https://github.com/tiangolo/fastapi/issues/610, it seems that FastAPI does not support query string flags `?features`. Another thing t... [07:45:14] (03CR) 10Kevin Bazira: [C: 03+2] Update the ores submodule to deploy deprecation message changes [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/941909 (https://phabricator.wikimedia.org/T342118) (owner: 10Kevin Bazira) [07:52:41] (03CR) 10Kevin Bazira: [V: 03+2 C: 03+2] Update the ores submodule to deploy deprecation message changes [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/941909 (https://phabricator.wikimedia.org/T342118) (owner: 10Kevin Bazira) [08:25:22] 10Machine-Learning-Team: Add deprecation messages for features not supported in ores-legacy - https://phabricator.wikimedia.org/T342663 (10achou) The `model_info` query parameter is also available in the endpoints `/v3/scores/` in ORES (see https://ores.wikimedia.org/v3#!/scoring/get_v3_scores) The result from... [08:42:41] (03PS1) 10AikoChou: ores-legacy: add error messages for features not supported [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) [08:46:25] 10Machine-Learning-Team, 10Patch-For-Review: [ores-legacy] Inconsistency when returning features - https://phabricator.wikimedia.org/T342791 (10achou) a:03achou [08:46:56] 10Machine-Learning-Team, 10Patch-For-Review: Add deprecation messages for features not supported in ores-legacy - https://phabricator.wikimedia.org/T342663 (10achou) a:03achou [08:56:49] (03CR) 10Klausman: [C: 03+1] ores-legacy: add features field to response models [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) (owner: 10AikoChou) [08:56:51] (03CR) 10Klausman: [C: 03+1] ores-legacy: add error messages for features not supported [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) (owner: 10AikoChou) [09:16:11] hi folks! [09:16:13] still in standup? [09:16:42] yep! [09:17:19] joining :) [09:29:18] https://meet.google.com/ikt-whmi-fqy?authuser=0 [09:29:23] argh, wrong URL [09:29:28] https://www.ibiblio.org/harris/500milemail.html [09:50:04] (03PS1) 10Kevin Bazira: Update the ores submodule to deploy fix of scorer deprecation message [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/941910 (https://phabricator.wikimedia.org/T342118) [09:58:01] (03CR) 10Ilias Sarantopoulos: [C: 03+1] Update the ores submodule to deploy fix of scorer deprecation message [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/941910 (https://phabricator.wikimedia.org/T342118) (owner: 10Kevin Bazira) [09:58:07] (03CR) 10Elukey: [C: 03+1] Update the ores submodule to deploy fix of scorer deprecation message [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/941910 (https://phabricator.wikimedia.org/T342118) (owner: 10Kevin Bazira) [10:06:58] (03CR) 10Kevin Bazira: [V: 03+2 C: 03+2] "Thanks for the reviews :)" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/941910 (https://phabricator.wikimedia.org/T342118) (owner: 10Kevin Bazira) [10:12:33] kevinbazira: if the cloud ores instance looks good we can deploy anytime :) [10:13:37] Yep, the scorer on beta has been fixed now: https://ores-beta.wmflabs.org/ui [10:14:54] kevinbazira: you rock, let's do it! [10:15:57] https://wikitech.wikimedia.org/wiki/ORES/Deployment#Production_cluster_(ores.wikimedia.org) [10:24:24] all the prechecks look fine on: deployment.eqiad.wmnet [10:24:28] deploying now [10:28:12] kevinbazira: not sure if you did it but next time check grafana first, to see if anything is ongoing or not etc.. [10:36:14] https://ores.wikimedia.org/ updateD! [10:36:18] nice work kevinbazira [10:36:57] Thanks for your help with this elukey! :) [10:48:58] * elukey lunch! [10:51:25] same [10:56:21] Me too! [11:29:04] 10Machine-Learning-Team, 10Add-Link, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Trizek-WMF) Everything works except : * [[https://kg.wikipedia.org/w/index.php?title=Sp... [11:29:58] 10Machine-Learning-Team, 10Add-Link, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Trizek-WMF) [11:32:03] 10Machine-Learning-Team, 10Add-Link, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Trizek-WMF) [11:49:18] 10Machine-Learning-Team, 10Add-Link, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Urbanecm_WMF) Thanks @Trizek-WMF, updated the final deployment patch to reflect that. [12:05:56] Day 4 of swarming [12:06:13] Swarming is a hard work to pronounce [12:28:09] (03PS2) 10AikoChou: ores-legacy: add error messages for features not supported [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) [12:29:48] o/ Chris [12:30:13] especially if you do it repeatedly. swarm swarm swarm swarm [12:35:55] elukey: klausman: these are the v1 endpoints in kserve -> https://github.com/kserve/kserve/blob/9b917d8349e4885abe4621db47def0a385ecd76d/python/kserve/kserve/protocol/rest/v1_endpoints.py#L87 [12:36:13] only explain and predict Β―\_(ツ)_/Β― [12:36:26] dang. [12:37:02] oups there is a models func. I'm gonna check what this does [12:46:07] yep exactly, I don't think we can customize [12:55:42] we can maintain our own fork of kserve and allow many endpoints ...NOT [12:56:04] maybe we can send a pull request if we feel so [12:56:38] but it should be for a generic use case or for an endpoint that can be configurable [12:57:34] I think we can use explain: and figure out what to do in the future [13:03:41] ok, I checked and the other functions are for v2. I think we can open an issue/pull request to add a metadata/info endpoint. I can look into it next week and open a task https://github.com/kserve/kserve/blob/9b917d8349e4885abe4621db47def0a385ecd76d/python/kserve/kserve/protocol/rest/server.py#L94 [13:05:01] isaranto: on discord they asked to me if we are planning to add new scores (like revert risk etc..) to https://en.wikipedia.org/w/api.php?action=query&list=recentchanges&rcprop=ids|oresscores&rclimit=50&utf8=1 [13:05:27] I was surprised to see the recent changes stream with the ores scores [13:05:54] do you think that it gets the results from the ores extension table? [13:06:46] Since Kserve is written in GO, I'll have a look [13:07:25] klausman: the controllers are written in go, not the library that we use (that is written in python) [13:07:30] two separate things [13:07:48] ah, righto. Then hacking in extra handlers might be even easier than I had thought [13:13:03] isaranto: o/ when you have a moment, can you review https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/942320 and https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/942063 [13:21:34] (03CR) 10Elukey: ores-legacy: add error messages for features not supported (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) (owner: 10AikoChou) [13:23:24] (03CR) 10Elukey: "Aiko I am ignorant about what this change does since I don't know exactly how fastapi works in this case, do you mind to add a bit more co" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) (owner: 10AikoChou) [13:23:34] aiko: done! Asked some follow ups, but it looks good [13:24:59] aiko: I am reviewing1 [13:26:39] elukey: probably it gets the results from there. on which discord? [13:26:59] there is a channel for 𝗦π—ͺπ—©π—Άπ—²π˜„π—²π—Ώ [13:27:01] 10Lift-Wing, 10Machine-Learning-Team: Enable batch inference for revscoring models - https://phabricator.wikimedia.org/T342555 (10achou) 05Openβ†’03Declined After investigation, we found that using batching for revscoring models does not provide much improvement in latency. Therefore, I'm going to close this... [13:27:16] ttps://discord.gg/JEXNCNHK63 [13:27:48] I never seen that api call, but it must be from the ores tables [13:33:21] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "A little more info need. other than that LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) (owner: 10AikoChou) [13:34:11] elukey: thanks! I'll add more context [13:40:41] (03CR) 10Ilias Sarantopoulos: ores-legacy: add error messages for features not supported (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) (owner: 10AikoChou) [13:41:17] aiko: the features patch is ok, just add some info. for the other one I adde some suggestions. thanks! [13:56:28] filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/942427 for the extra resources to the envoy proxy [13:56:42] with the 50 rev-ids limit we should be ok in theory [14:02:18] I am investigating an issue with eswikiquote (eswikibooks is fine!) [14:02:36] we dont get new predictions for quote [14:04:01] let us know if you need help [14:06:51] lol pod is down [14:07:01] (not really lol πŸ˜›) [14:07:48] and it may have never been up (?). I remember everything was running after I deployed [14:07:59] `Init:CrashLoopBackOff` [14:09:13] figured it out, it was the model /storage initializer [14:09:22] issuing a fix now [14:17:09] my bad. I was focused on wikibooks while the other was failing miserably [14:20:19] just deployed [14:22:01] all good now, will check on the wiki side [14:25:38] all good, new revisions in! I want a cookie for this failure! [14:25:45] also , I need alerts sooo bad [14:29:00] :D [14:29:16] I tested the new settings for the envoy proxy in ores legacy, not enough [14:29:18] sigh [14:30:07] we can probably limit the batch to say 10/20 at this point [14:30:12] I'll find a sweet spot [14:30:35] I agree. otherwise the request takes too long as well [14:31:33] isaranto: wait, why did my tests work, then?> [14:34:16] klausman: I think you may have targeted eswikiquotewiki (the one we are going to deprecate) instead of eswikiquote [14:35:39] dammit [14:36:39] it takes two to tango <3 [14:37:15] and you even had a note in the paste! [14:37:21] what worried me is the fact that no errors appeared on logstash [14:39:31] so was this both goodfaith and damaging? [14:40:48] 10Machine-Learning-Team, 10Add-Link, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Urbanecm_WMF) Per a [Slack discussion](https://wikimedia.slack.com/archives/G0101329ZC7/... [14:41:00] 10Machine-Learning-Team, 10Add-Link, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Urbanecm_WMF) [14:44:07] 10Machine-Learning-Team, 10ORES: Users/UserAgents that use ORES - https://phabricator.wikimedia.org/T342835 (10isarantopoulos) In [[ https://turnilo.wikimedia.org/#webrequest_sampled_128/4/N4IgbglgzgrghgGwgLzgFwgewHYgFwgDmAThACYgA0408SqGOAygKZobaFT7YwILUMAWxbIcLfCACiaAMYB6AKoAVAMJUQAMwgI0LYtzwBtUGgCeABwkFhE6sR... [14:44:18] the issue was only with goodfaith [14:44:50] 10Machine-Learning-Team, 10Add-Link, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Growth-Team (Current Sprint), and 2 others: Deploy "add a link" to 10th round of wikis - https://phabricator.wikimedia.org/T308135 (10Urbanecm_WMF) a:05Trizek-WMFβ†’03Urbanecm_WMF I'll do the deployment on Tuesday. [14:44:56] Ok, retesting that, then [14:46:04] Looks good. About 40qps at 10 threads [14:47:08] 63qps @20 threads [14:47:15] No errors [14:47:27] This was with: [14:47:31] 2023/07/27 14:45:56 URL: https://inference.svc.eqiad.wmnet:30443/v1/models/eswikiquote-goodfaith:predict [14:47:34] 2023/07/27 14:45:56 Host: eswikiquote-goodfaith.revscoring-editquality-goodfaith.wikimedia.org [14:47:45] nice, thanks Tobias! [14:50:06] (03PS3) 10AikoChou: ores-legacy: add error messages for features not supported [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) [14:55:37] (03CR) 10AikoChou: ores-legacy: add error messages for features not supported (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) (owner: 10AikoChou) [14:56:26] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "Nice! Thanks for the unit test!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) (owner: 10AikoChou) [15:59:34] (03PS3) 10AikoChou: ores-legacy(fix): add features in pydantic model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) [16:01:40] (03CR) 10AikoChou: [C: 03+2] "Thanks for the review :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) (owner: 10AikoChou) [16:01:47] (03CR) 10Ilias Sarantopoulos: ores-legacy(fix): add features in pydantic model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) (owner: 10AikoChou) [16:02:33] (03Merged) 10jenkins-bot: ores-legacy: add error messages for features not supported [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942320 (https://phabricator.wikimedia.org/T342663) (owner: 10AikoChou) [16:03:53] (03PS4) 10AikoChou: ores-legacy(fix): add features in pydantic model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) [16:06:23] (03CR) 10AikoChou: [C: 03+2] "Thanks for the review :D" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) (owner: 10AikoChou) [16:07:16] (03Merged) 10jenkins-bot: ores-legacy(fix): add features in pydantic model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/942063 (https://phabricator.wikimedia.org/T342791) (owner: 10AikoChou) [16:14:44] going afk folks! [16:14:45] o/ [16:19:34] \o [16:19:40] o/ [16:28:23] 10Machine-Learning-Team, 10ORES: Users/UserAgents that use ORES - https://phabricator.wikimedia.org/T342835 (10isarantopoulos) The following bots/User agents are actively using ORES - https://dashboard.wikiedu.org/ is using ORES https://github.com/WikiEducationFoundation/WikiEduDashboard . There references ar... [16:30:01] ciao Luca! [16:30:31] I added some info in the above and I think we can tackle some things tomorrow! [16:47:29] We are live in hewiki and itwiki. Thanks once again Amir1: for the deployments!! [16:47:39] logging off folks, cu tomorrow! [16:48:30] yay thanks Amir1 \o/ [16:48:32] ciao Ilias! [16:48:49] wohooo πŸŽ‰