[07:35:29] hi folks! [07:35:36] will start a little later today, ttl! [09:34:55] back! [09:35:04] Going to roll restart ORES for security updates [09:40:21] (03PS1) 10AikoChou: outlink: log response returned from MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/855963 (https://phabricator.wikimedia.org/T320374) [10:04:16] this is the pause container's code, there is a hidden gem on it [10:04:17] https://github.com/kubernetes/kubernetes/blob/master/build/pause/linux/pause.c#L67 [10:19:10] 10Machine-Learning-Team, 10Research: Upload new outlinks topic model to LiftWing - https://phabricator.wikimedia.org/T322881 (10achou) a:03achou [10:40:51] 10Lift-Wing, 10Machine-Learning-Team, 10Research: Upload new outlinks topic model to LiftWing - https://phabricator.wikimedia.org/T322881 (10achou) [10:45:04] (03CR) 10Elukey: outlink: log response returned from MW API (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/855963 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [11:17:55] I managed to upload the new outlink topic model at 11:11:11 today :D :D [11:26:15] :D [11:27:26] * elukey lunch! [12:19:36] Going food hunting for a bit :) [13:15:44] 10Lift-Wing, 10Machine-Learning-Team, 10Research: Upload new outlinks topic model to LiftWing - https://phabricator.wikimedia.org/T322881 (10achou) Thanks for updating the model card! I've tested a model server with the new model locally and the model seems to be working great :) The new outlinks topic mode... [14:03:24] elukey: o/ [14:04:22] elukey: since we use resp.json() at https://github.com/mediawiki-utilities/python-mwapi/blob/master/mwapi/async_session.py#L88 [14:04:46] elukey: we read response's body as json (https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientResponse.json) [14:05:51] elukey: looks like we're not able to get other attributes in the ClientResponse. I tried logging.info(r.status) but I got "'dict' object has no attribute 'status'" [14:10:53] elukey: I feel we need to change async_session.py somehow.. [14:25:19] (03PS2) 10AikoChou: outlink: log response returned from MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/855963 (https://phabricator.wikimedia.org/T320374) [14:26:48] aiko: ahh okok nevermind them.. do you mean that we get a dict with all the fields? [14:27:08] if so maybe r['status'] works? [14:34:19] elukey: not all the fields, we only got response's body (outlinks in this case) [14:35:03] aiko: okok got it, so we don't know the return code basically [14:35:31] r['status'] .. doesn't work (keyError) [14:35:36] elukey: yep [14:35:57] ok let's try with what we have, and if it is not enough we'll add a feature to mwapi [14:36:00] does it sound good? [14:36:35] elukey: ok! sounds good to me [14:37:23] (03CR) 10AikoChou: outlink: log response returned from MW API (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/855963 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [14:49:24] (03CR) 10Elukey: [C: 03+1] "Let's try :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/855963 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [14:58:39] (03PS3) 10AikoChou: outlink: log response returned from MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/855963 (https://phabricator.wikimedia.org/T320374) [15:10:35] elukey: ---^ I added a logging.debug to log every input and the number of outlinks fetched.. so I can compare them with problematic inputs [15:11:52] elukey: qq - if I want to change the log level to debug or change back, is it true I need to submit a patch to modify the KSERVE_LOGLEVEL env on value.yaml every time and deploy again? or is there a simple way to change env variable for the pod? [15:14:12] aiko: for testing use cases I can edit manually and add the env variable, the pods gets re-created and the we can revert back when finished [15:14:37] (03CR) 10Elukey: [C: 03+1] outlink: log response returned from MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/855963 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [15:15:25] elukey: I see, thanks :) [15:15:59] (03CR) 10AikoChou: [C: 03+2] outlink: log response returned from MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/855963 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [15:17:35] (03Merged) 10jenkins-bot: outlink: log response returned from MW API [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/855963 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [15:46:48] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/855995 :D [15:47:00] aiko: for the moment the outlink config is fine, but we should probably avoid to have multiple version tags [15:48:21] (I mean the alloutlinks_202209 etc..) [15:48:30] not something to be solved now, but let's think about it [15:48:45] elukey: hmm so don't change it for now? [15:49:25] aiko: nono I am going to merge as soon as CI gives the +2 [15:49:39] but it feels a little confusing to have our timestamp and the alloutlinks_date version [15:50:01] (merged) [15:54:33] yeah a little confusing [16:04:49] elukey: I just deployed to staging.. could you help me change the log level manually? [16:05:04] aiko: 5 euros! [16:05:06] :D [16:05:16] what do you need? DEBUG? [16:05:36] yes DEBUG haha [16:06:00] the transformer right? [16:06:14] yep [16:07:13] aiko: pod is being created [16:07:53] elukey: niceeee! thank you <3 [16:22:41] aiko: I may have found a way in the revscoring extractor to pass the cache, so that if we use extended_features=true we don't fully recompute [16:22:44] see [16:22:47] [I 221111 16:21:43 decorators:35] Function fetch_features took 1.0253 seconds to execute. [16:22:50] [I 221111 16:21:43 decorators:35] Function fetch_features took 0.1758 seconds to execute. [16:23:00] the second one is not as heavy as the former [16:24:47] but I'll need your help to figure out if it is sound [16:24:47] wow... that's great! you use the parameter "cache"? [16:24:56] yeah [16:25:23] and we end up in https://github.com/wikimedia/revscoring/blob/master/revscoring/extractors/api/extractor.py#L313 [16:25:50] I hope that the "Dependents" fields are the same in both use cases [16:25:58] well I can check with some tests probably [16:27:01] may need to check if the feature values returned are the same as before [16:29:02] in my head if the cache is the same with/without the trimmed values it should be ok right? [16:34:37] what's the data you passed to the cache? [16:34:37] https://docs.python.org/3/library/stdtypes.html#mapping-types-dict [16:34:44] "Dictionaries compare equal if and only if they have the same (key, value) pairs (regardless of ordering). Order comparisons (‘<’, ‘<=’, ‘>=’, ‘>’) raise TypeError." [16:34:49] nice so I can just == in theory [16:36:31] mmm I got False sigh [16:38:00] ufff [16:38:28] ahh I didn't mean that [16:39:16] yes yes you meant the list of features returned, before/after [16:39:20] I'll try to check it as well [16:39:25] fingers crossed [16:39:53] yeah with/without cache [16:43:23] aiko: the == comparison of the feature list is true [16:43:32] with/without cache [16:44:23] I tried this https://phabricator.wikimedia.org/P39276 [16:44:56] perfect :D [16:58:33] but it seems not working in the process pool (I think because of the serialization etc..) [17:01:30] nooooo :'( [17:05:32] I have a workaround, testing it [17:21:22] [E 221111 17:08:31 outlink_transformer:91] KeyError occurs for Battle_of_Pisky_(2022) (en). Reason: 'query'. MW API returns: {'batchcomplete': True} [17:21:22] [D 221111 17:08:31 outlink_transformer:94] en Battle_of_Pisky_(2022) fetched 463 outlinks [17:22:27] So it’s not an issue at all! we actually have fetched data. It's MW API sometimes returns {'batchcomplete': True} at the end of async_generator object [17:22:59] nice :) [17:25:36] so it seems that the local cache is not good to be pickled, I get exceptions when trying to pass it back [17:25:40] sigh [17:25:44] I'll restart next week [17:25:49] hopefully there is a solution [17:25:55] have a good weekend folks! [17:26:49] ack! have a nice weekend Luca :) [17:37:34] \o [17:44:00] have a great weekend /o