[05:47:14] <wikibugs>	 (03PS2) 10Ilias Sarantopoulos: llm: load test langid model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/967220 (https://phabricator.wikimedia.org/T340507)
[06:05:40] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [V: 03+2 C: 03+2] llm: load test langid model [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/967220 (https://phabricator.wikimedia.org/T340507) (owner: 10Ilias Sarantopoulos)
[06:22:17] <elukey>	 morning folks
[06:22:52] <elukey>	 kevinbazira: o/ before starting, did you notice a reduction in total time (if we have it in the logs) with 250 instead of 500?
[06:23:34] <kevinbazira>	 elukey: o/ nope there was no noticable reduction in time.
[06:24:41] <elukey>	 ah snap then this is something weird
[06:24:56] <elukey>	 changing it to 100, but doesn't seem good
[06:25:06] <kevinbazira>	 yep, the rec-api query still hangs and doesn't return results
[06:26:24] <elukey>	 MAX_CANDIDATES set to 100, pod is spinning up
[06:27:29] <elukey>	 ready to test :)
[06:27:59] <kevinbazira>	 thanks, testing now ..
[06:29:34] <kevinbazira>	 same results:
[06:29:43] <kevinbazira>	 ```
[06:29:43] <kevinbazira>	 $ time curl 'https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/?s=en&t=fr&n=3&article=Apple'
[06:29:43] <kevinbazira>	 upstream connect error or disconnect/reset before headers. reset reason: connection termination
[06:29:44] <kevinbazira>	 real	0m48.484s
[06:29:44] <kevinbazira>	 user	0m0.008s
[06:29:44] <kevinbazira>	 sys	0m0.007s
[06:29:45] <kevinbazira>	 ```
[06:31:37] <kevinbazira>	 container logs:
[06:31:45] <kevinbazira>	 ```
[06:31:45] <kevinbazira>	 Fri Oct 20 06:28:53 2023 - HARAKIRI [core 0] 127.0.0.1 - GET /api/?s=en&t=fr&n=3&article=Apple since 1697783317
[06:31:45] <kevinbazira>	 Fri Oct 20 06:28:53 2023 - HARAKIRI !!! end of worker 1 status !!!
[06:31:45] <kevinbazira>	 2023-10-20 06:28:53,941 recommendation.utils.event_logger log_api_request():39 INFO -- Logging event: {"schema": "TranslationRecommendationAPIRequests", "$schema": "/analytics/legacy/$translationrecommendationapirequests/1.0.0", "revision": 16261139, "event": {"timestamp": 1697783333, "sourceLanguage": "en", "targetLanguage": "fr", "seed": "Apple", "searchAlgorithm": "morelike"}, "webHost": 
[06:31:45] <kevinbazira>	 "recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443", "client_dt": "2023-10-20T06:28:53.940930", "meta": {"stream": "eventlogging_TranslationRecommendationAPIRequests", "domain": "recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443"}}
[06:31:46] <kevinbazira>	 2023-10-20 06:28:54,450 recommendation.api.types.translation.candidate_finders get_morelike_candidates():39 INFO -- morelike returned 101 results
[06:31:46] <kevinbazira>	 DAMN ! worker 1 (pid: 134) died, killed by signal 9 :( trying respawn ...
[06:31:47] <kevinbazira>	 Respawned uWSGI worker 1 (new pid: 159)
[06:31:48] <kevinbazira>	 ```
[06:34:29] <elukey>	 from the logs it is not clear though if it bottleneck is the candidates number though
[06:34:55] <elukey>	 I don't see a clear timing, like "start of request -> etc.."
[06:37:21] <isaranto>	 morning!
[06:41:25] <elukey>	 kalimera :
[06:41:27] <elukey>	 :)
[06:42:38] <elukey>	 kevinbazira: I checked with nsenter on the pods, and `netstat -tunap` (that shows all the TCP conns etc..) has these weird entries
[06:42:41] <elukey>	 tcp6       0      1 2620:0:860:302:cd:33276 2620:0:860:ed1a::1:443  SYN_SENT    3248538/uwsgi       
[06:42:44] <elukey>	 tcp6       0      1 2620:0:860:302:cd:33266 2620:0:860:ed1a::1:443  SYN_SENT    3248538/uwsgi       
[06:42:47] <elukey>	 tcp6       0      1 2620:0:860:302:cd:33256 2620:0:860:ed1a::1:443  SYN_SENT    3248538/uwsgi  
[06:43:01] <elukey>	 all the other sockets are either ESTABLISHED (connection ok) or TIME_WAIT (connection already completed)
[06:43:20] <elukey>	 SYN_SENT means that the client started the conn, but didn't get an answer yet
[06:43:35] <elukey>	 so I think that this may be the issue, we are hanging waiting for something
[06:44:15] <elukey>	 and if I do a reverse dns lookup, I get text-lb.codfw.wikimedia.org
[06:44:43] <elukey>	 that is our fronted load balancer, so it seems that we are trying to connect to a wikimedia.org endpoint 
[06:45:01] <elukey>	 (hanging, since it is now allowed, we need to use the envoy proxy)
[06:45:47] <kevinbazira>	 I thought we took care of the envoy proxy issues o_o
[06:47:15] <elukey>	 maybe there is still a sneaky one that lingers 
[06:50:45] <kevinbazira>	 does this mean the rec-api is not able to access the  wikimedia.org endpoint through the envoy proxy?
[06:53:12] <elukey>	 yes I think that's it, I didn't see the "pageviews" setting, it is not right
[06:53:25] <elukey>	 localhost:6500 is only for the PHP API
[06:53:37] <elukey>	 not sure what endpoint "pageviews" need
[06:54:49] <elukey>	 ahh snap
[06:54:50] <elukey>	 query = ${endpoints:pageviews}/per-article/{source}.wikipedia/all-access/user/{title}
[06:55:58] <elukey>	 yes ok it tries to access https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews#Pageview_counts_by_article
[06:56:25] <elukey>	 that hits restbase IIRC
[06:56:52] <elukey>	 I don't recall if we have an internal endpoint, lemme think
[06:57:12] <elukey>	 it hits the AQS service, not sure if with the same URL
[06:58:12] <elukey>	 so the flow should be
[06:58:39] <elukey>	 LB (wikimedia.org) -> Restbase (proxy/cache) -> AQS LB -> AQS service
[06:59:39] <elukey>	 kevinbazira: we have the aqs listener in service proxy, port 6020
[07:00:48] <isaranto>	 sorry folks but sometimes when I switch to the irccloud tab I write sth and then past messages pop up . so some responses of mine may seem out of context
[07:01:02] <isaranto>	 probably I need to restart or sth
[07:01:37] <kevinbazira>	 elukey: great, so we should point the pageviews port to 6020
[07:03:09] <elukey>	 kevinbazira: in theory yes, but I am not 100% sure if the URLs are the same
[07:03:31] <elukey>	 the current URL is the external/public one, exposed by restbase
[07:03:41] <elukey>	 that proxies to our internal endpoint, AQS
[07:03:58] <elukey>	 IIRC the format is not the same
[07:04:04] <elukey>	 I am trying to hit the endpoint
[07:05:28] <kevinbazira>	 since you have access to the pod, is it possible to manually change the listener in the deployment conifg and port in recommendation/data/recommendation_liftwing.ini so we can test this without a gerrit patch?
[07:05:56] <elukey>	 kevinbazira: ah no that is too much magic, can't do it :(
[07:06:11] <kevinbazira>	 ok, should I push a patch for this?
[07:06:55] <elukey>	 kevinbazira: I think we need to follow up with Data Engineering first (they manage AQS) to figure out what is the format of the URLs of the internal API
[07:08:18] <elukey>	 atm we have pageviews = http://localhost:6500/api/rest_v1/metrics/pageviews
[07:08:51] <elukey>	 elukey@stat1004:~$ curl http://aqs.discovery.wmnet:7232/api/rest_v1/metrics/pageviews
[07:08:54] <elukey>	 {"type":"https://mediawiki.org/wiki/HyperSwitch/errors/not_found#route","title":"Not found.","method":"get","uri":"/api/rest_v1/metrics/pageviews"}
[07:08:57] <elukey>	 kevinbazira: --^
[07:09:17] <kevinbazira>	 yes, that was configured based on: https://github.com/wikimedia/research-recommendation-api/blob/3c50d88504e66fd897416374d64235c3c1733234/recommendation/data/recommendation.ini#L3C1-L3C1
[07:09:22] <elukey>	 if you have time you can connect to #wikimedia-analytics and ask to the data engineering folks
[07:09:35] <kevinbazira>	 sure, let me follow up
[07:09:43] <elukey>	 I need to run an errand, bbiab
[08:08:08] <isaranto>	 Afk commuting
[08:50:57] <elukey>	 back :)
[08:56:34] <elukey>	 kevinbazira: how is it going?
[08:57:42] <kevinbazira>	 elukey: haven't got a response yet. I actually tagged you in the question in #wikimedia-analytics 
[09:04:28] <elukey>	 ok IRC?
[09:04:32] <elukey>	 *on
[09:04:47] <elukey>	 I don't see it :(
[09:12:32] <kevinbazira>	 Here is the screenshot of the message. not sure why you're not seeing it since I tagged you ... https://usercontent.irccloud-cdn.com/file/dhRZgkiu/Screenshot%20from%202023-10-20%2013-07-43.png
[09:15:31] <elukey>	 kevinbazira: I see that you joined but not msg delivered
[09:16:50] <kevinbazira>	 I've sent it again.
[09:17:01] <kevinbazira>	 are you able to see it?
[09:17:58] <isaranto>	 for some reason it wasnt sent previously. now it is there!
[09:17:59] <elukey>	 now yes
[09:18:33] <kevinbazira>	 okok
[09:18:42] <elukey>	 kevinbazira: when you explain on IRC try to add more info about what your goal is, so it will be easier for people to understand (like, what is the listener, etc..)
[09:19:06] <elukey>	 (not everybody worked with it)
[09:19:43] <elukey>	 in this case we know we want to use the AQS endpoint though, the main issue is to find the right URI
[09:27:05] <elukey>	 I may have something
[09:35:17] <wikibugs>	 (03PS1) 10Elukey: Fix pageviews base endpoint [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/967401 (https://phabricator.wikimedia.org/T348607)
[09:35:23] <elukey>	 kevinbazira: --^
[09:35:27] <elukey>	 I think it should work
[09:36:09] <elukey>	 so we have to
[09:36:14] <elukey>	 1) rebuild the docker image
[09:36:28] <elukey>	 2) change the docker image and add the aqs listener in deployments-charts
[09:36:58] <wikibugs>	 (03CR) 10Kevin Bazira: [C: 03+1] "ok, let's give this a shot" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/967401 (https://phabricator.wikimedia.org/T348607) (owner: 10Elukey)
[09:37:15] <kevinbazira>	 elukey: ok, let's give this a shot
[09:38:53] <kevinbazira>	 will push a patch for the deployments-charts once the new image has been built
[09:40:22] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Fix pageviews base endpoint [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/967401 (https://phabricator.wikimedia.org/T348607) (owner: 10Elukey)
[09:41:14] <elukey>	 isaranto: if you have time later on https://gerrit.wikimedia.org/r/c/research/recommendation-api/+/966543
[09:42:00] <isaranto>	 ack, will do!
[09:53:21] <elukey>	 <3
[09:53:35] <elukey>	 kevinbazira: +1ed the change, let's test! (if you have time, otherwise later)
[09:55:02] <kevinbazira>	 sure, merging now so we can test ...
[09:56:52] <kevinbazira>	 deploying on staging ...
[09:57:32] <elukey>	 ack
[09:57:39] <elukey>	 filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/967408 for the SLO metrics calculations
[09:57:42] <elukey>	 hope that it will work
[10:01:12] <kevinbazira>	 elukey, the api is returning results now :)  
[10:01:20] <kevinbazira>	 ```
[10:01:20] <kevinbazira>	 $ time curl 'https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/?s=en&t=fr&n=3&article=Apple'
[10:01:20] <kevinbazira>	 [{"pageviews": 0, "title": "Beetroot", "wikidata_id": "Q99548274", "rank": 243.0}, {"pageviews": 0, "title": "Malus_\u00d7_zumi", "wikidata_id": "Q5990804", "rank": 239.0}, {"pageviews": 0, "title": "PRI_disease_resistant_apple_breeding_program", "wikidata_id": "Q19840594", "rank": 237.0}]
[10:01:20] <kevinbazira>	 real	0m2.658s
[10:01:20] <kevinbazira>	 user	0m0.013s
[10:01:20] <kevinbazira>	 sys	0m0.000s
[10:01:21] <kevinbazira>	 ```
[10:01:34] <isaranto>	 🎉great work both!
[10:01:51] <klausman>	 agreed
[10:04:13] <elukey>	 yesssssss
[10:05:17] <elukey>	 kevinbazira: we could test with 500 quickly to check how fast it is
[10:05:30] <elukey>	 and possibly also with more workers, just to find a sweet spot for one pod
[10:05:34] <elukey>	 what do you think?
[10:05:39] <kevinbazira>	 okok, are you going to do it manually or should I push a patch?
[10:05:55] <elukey>	 manually yes, is it ok now? Or is it lunch time for you?
[10:06:15] <kevinbazira>	 no it's fine, let's not break this momentum :)
[10:07:16] <elukey>	 kevinbazira: ack, new pod is up with 500
[10:07:57] <kevinbazira>	 testing ...
[10:08:17] <kevinbazira>	 results:
[10:08:17] <kevinbazira>	 ```
[10:08:17] <kevinbazira>	 $ time curl 'https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/?s=en&t=fr&n=3&article=Apple'
[10:08:17] <kevinbazira>	 [{"pageviews": 0, "title": "Peanut_stunt_virus", "wikidata_id": "Q7157945", "rank": 496.0}, {"pageviews": 0, "title": "Kentville_Research_and_Development_Centre", "wikidata_id": "Q4816432", "rank": 495.0}, {"pageviews": 0, "title": "Specific_replant_disease", "wikidata_id": "Q889601", "rank": 494.0}]
[10:08:18] <kevinbazira>	 real	0m2.950s
[10:08:18] <kevinbazira>	 user	0m0.010s
[10:08:19] <kevinbazira>	 sys	0m0.005s
[10:08:19] <kevinbazira>	 ```
[10:09:21] <elukey>	 I tested as well from stat1004, I get something in the range of 2.2~2.7 seconds
[10:09:36] <elukey>	 so in theory it seems that the max-candidates number is not that heavy 
[10:10:04] <kevinbazira>	 yep, I agree
[10:10:36] <elukey>	 we could try to bump the workers + cpus, maybe getting up to 4, what do you think?
[10:10:46] <kevinbazira>	 +1
[10:11:05] <elukey>	 after that we'll have to set up a small load test, to have better comparisons
[10:11:11] <elukey>	 but we can do a quick test now
[10:12:25] * klausman lunch
[10:16:22] <elukey>	 kevinbazira: done!
[10:16:30] <elukey>	 from a quick test I don't see a big difference
[10:17:21] <kevinbazira>	 testing ...
[10:17:36] <kevinbazira>	 results:
[10:17:36] <kevinbazira>	 ```
[10:17:36] <kevinbazira>	 $ time curl 'https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/?s=en&t=fr&n=3&article=Apple'
[10:17:36] <kevinbazira>	 [{"pageviews": 0, "title": "Peanut_stunt_virus", "wikidata_id": "Q7157945", "rank": 496.0}, {"pageviews": 0, "title": "Kentville_Research_and_Development_Centre", "wikidata_id": "Q4816432", "rank": 495.0}, {"pageviews": 0, "title": "Specific_replant_disease", "wikidata_id": "Q889601", "rank": 494.0}]
[10:17:37] <kevinbazira>	 real	0m2.141s
[10:17:37] <kevinbazira>	 user	0m0.010s
[10:17:38] <kevinbazira>	 sys	0m0.005s
[10:17:38] <kevinbazira>	 ```
[10:17:50] <kevinbazira>	 elukey: yep, not much is changing.
[10:19:14] <elukey>	 kevinbazira: we could think about keeping 2 cpus and 2 workers, with 500 max candidates, at least for the moment. We rollout the change to staging and prod via deployment-charts, so we are in a kind-of stable state
[10:19:42] <elukey>	 then I'd say that we could do a little load test in staging, with more URLs (different than the Apple one)
[10:20:08] <elukey>	 what do you think?
[10:20:42] <kevinbazira>	 +1 on 2 cpus and 2 workers with 500 max candidates. let me push a patch for that.
[10:22:01] <kevinbazira>	 results on testing a different article other than Apple:
[10:22:01] <kevinbazira>	 ```
[10:22:01] <kevinbazira>	 $ time curl 'https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/?s=en&t=fr&n=3&article=Tennis'
[10:22:01] <kevinbazira>	 [{"pageviews": 0, "title": "2017_Australian_Open_\u2013_Men's_singles_final", "wikidata_id": "Q28600417", "rank": 499.0}, {"pageviews": 0, "title": "ATP_rankings", "wikidata_id": "Q1571500", "rank": 495.0}, {"pageviews": 0, "title": "PointTracker", "wikidata_id": "Q7207919", "rank": 491.0}]
[10:22:01] <kevinbazira>	 real	0m2.776s
[10:22:01] <kevinbazira>	 user	0m0.010s
[10:22:02] <kevinbazira>	 sys	0m0.005s
[10:22:02] <kevinbazira>	 ```
[10:30:48] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C: 03+1] "LGTM! I just left a comment with an alternative suggestion for skipping pep8 checks. Feel free to skip it if you don't like it." [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey)
[10:40:22] <wikibugs>	 (03CR) 10Elukey: Fix pre-commit errors and bump version (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey)
[10:43:14] <kevinbazira>	 deployment on staging is done.
[10:43:15] <kevinbazira>	 results:
[10:43:15] <kevinbazira>	 ```
[10:43:15] <kevinbazira>	 $ time curl 'https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/?s=en&t=fr&n=3&article=Basketball'
[10:43:15] <kevinbazira>	 [{"pageviews": 0, "title": "2019_United_States_FIBA_Basketball_World_Cup_team", "wikidata_id": "Q56042822", "rank": 495.0}, {"pageviews": 0, "title": "Molly_Bolin", "wikidata_id": "Q27451749", "rank": 482.0}, {"pageviews": 0, "title": "Dom_Flora", "wikidata_id": "Q5289728", "rank": 478.0}]
[10:43:15] <kevinbazira>	 real	0m2.902s
[10:43:16] <kevinbazira>	 user	0m0.009s
[10:43:16] <kevinbazira>	 sys	0m0.005s
[10:43:17] <kevinbazira>	 ```
[10:43:17] <kevinbazira>	 going to deploy to prod now ...
[10:44:02] <elukey>	 super
[10:44:47] <elukey>	 kevinbazira: we'll need to perform a more precise load test with something like wrk, to have a good idea about how many conns we can sustain etc.. nothing major, but basically like what Aiko and Ilias did for inference-services (in the repo test dir I mean)
[10:45:21] <kevinbazira>	 sure sure
[10:51:16] <elukey>	 going afk for lunch!
[10:51:32] <kevinbazira>	 deployment to both eqiad and codfw is done.
[10:51:33] <kevinbazira>	 results:
[10:51:33] <kevinbazira>	 ```
[10:51:33] <kevinbazira>	 $ time curl "https://recommendation-api-ng.discovery.wmnet:31443/api/?s=en&t=fr&n=3&article=Basketball"
[10:51:33] <kevinbazira>	 [{"pageviews": 0, "title": "2019_United_States_FIBA_Basketball_World_Cup_team", "wikidata_id": "Q56042822", "rank": 495.0}, {"pageviews": 0, "title": "Molly_Bolin", "wikidata_id": "Q27451749", "rank": 482.0}, {"pageviews": 0, "title": "Dom_Flora", "wikidata_id": "Q5289728", "rank": 478.0}]
[10:51:33] <kevinbazira>	 real	0m1.597s
[10:51:34] <kevinbazira>	 user	0m0.007s
[10:51:34] <kevinbazira>	 sys	0m0.007s
[10:51:35] <kevinbazira>	 ```
[10:51:35] <kevinbazira>	 prod is much faster than staging :)
[10:51:52] <elukey>	 weird, I wouldn't have expected that 
[10:52:02] <elukey>	 we have more pods but it shouldn't change much
[10:52:05] <elukey>	 mmmm
[11:13:25] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C: 03+1] Fix pre-commit errors and bump version (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey)
[11:20:19] <wikibugs>	 10Machine-Learning-Team: Configure envoy settings to enable rec-api-ng container to access endpoints external to k8s/LiftWing - https://phabricator.wikimedia.org/T348607 (10kevinbazira) We discovered that the [[ https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews#Pageview_counts_by_article | pageviews en...
[11:29:37] <wikibugs>	 10Machine-Learning-Team: Refactor LLM class and model server to run locally - https://phabricator.wikimedia.org/T349371 (10isarantopoulos)
[11:29:50] <wikibugs>	 10Machine-Learning-Team: Refactor LLM class and model server to run locally - https://phabricator.wikimedia.org/T349371 (10isarantopoulos)
[11:35:40] * isaranto lunch!
[11:43:42] <wikibugs>	 10Machine-Learning-Team: Investigate recommendation-api-ng internal endpoint failure - https://phabricator.wikimedia.org/T347475 (10kevinbazira) In T348607#9267956 we fixed the envoy listener for the pageviews endpoint. Now the rec-api-ng is able to access all external endpoints from k8s/LiftWing.  We also run l...
[11:46:40] <wikibugs>	 10Machine-Learning-Team: Investigate recommendation-api-ng internal endpoint failure - https://phabricator.wikimedia.org/T347475 (10kevinbazira) @isaac, the rec-api-ng endpoint now works as shown below. Please let us know whether there are edge cases we might have missed: ` $ time curl "https://recommendation-ap...
[11:59:03] <klausman>	 elukey: should I wait for Kamila's review of the Golang change? Also, Effie is out til Monday
[12:19:21] <elukey>	 klausman: nono you can go ahead, Janis signed off, it was to get somebody from service ops
[12:19:30] <klausman>	 righto
[12:26:23] <elukey>	 klausman: you need to kick off the build script from root on build2001
[12:26:33] <elukey>	 should be build-production-images etc..
[12:26:42] <klausman>	 already done :)
[12:26:44] <elukey>	 (and git pull the commit et.c.)
[12:26:45] <elukey>	 okok
[12:26:50] <klausman>	 == Step 2: publishing ==
[12:26:53] <klausman>	 == Build done! ==
[12:27:20] <elukey>	 did it say what image was published?
[12:27:21] <klausman>	 it's not visible yet on https://docker-registry.wikimedia.org/, so I suspect it's just some sync delay
[12:27:31] <elukey>	 yeah but can you pull it?
[12:28:08] <elukey>	 seems good yes, okok
[12:28:18] <elukey>	 good!
[12:28:19] <klausman>	 yes
[12:28:43] <klausman>	 And with that Yak shaven, I can now go back to the kserve update :D
[12:32:20] <klausman>	 which still can't fetch that very image I just built (and fetched :-/
[12:39:37] <wikibugs>	 (03PS7) 10Elukey: Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542
[12:39:45] <wikibugs>	 (03PS7) 10Elukey: Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543
[12:40:05] <wikibugs>	 (03CR) 10Elukey: Fix pre-commit errors and bump version (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey)
[12:40:15] <wikibugs>	 (03CR) 10Elukey: [V: 03+2 C: 03+2] Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 (owner: 10Elukey)
[12:42:42] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey)
[12:43:12] <elukey>	 all right rec-api-ng now with pre-commit support
[12:43:59] <wikibugs>	 (03Merged) 10jenkins-bot: Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey)
[12:44:38] <elukey>	 kevinbazira: I am going to update deployment-charts with the new image, since it uses the latest debian bullseye (so we fix some security concerns)
[12:47:20] <kevinbazira>	 sure sure
[12:53:14] <elukey>	 isaranto: created https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/967442/ to add the new env variable for revert risk (to merge before your patch for bullseye updates)
[12:53:24] <klausman>	 elukey: it looks like the image_tag thing breaks with the golang image name. I suspect because there is a dot in it
[12:53:41] <elukey>	 what is the image_tag thing?
[12:53:44] <klausman>	 If I use the naked image name (no {{ .. | }}), it works
[12:54:01] <klausman>	 the very thing you mentioned on the other patch as forgetting about it
[12:54:34] <elukey>	 sure but could you explain what is "works"? :)
[12:54:39] <elukey>	 local build, etc..?
[12:54:39] <klausman>	 `FROM {{ 'golang1.21' | image_tag }} as build` does not work
[12:54:48] <klausman>	 `FROM docker-registry.wikimedia.org/golang1.21 as build` works
[12:54:58] <klausman>	 and yes, local build.
[12:55:17] <elukey>	 what is the error?
[12:55:30] <elukey>	 do you have golang1.21 in your local docker?
[12:55:42] <elukey>	 I think that it needs to be there if you want to test it locally
[12:56:45] <klausman>	 Ok, so without golang present, I get:
[12:56:57] <klausman>	 2023-10-20 14:56:53,145 [docker-pkg-build] ERROR - Unexpected error building image docker-registry.wikimedia.org/kserve-build:0.11.1: Image docker-registry.wikimedia.org/golang1.21 not found (image.py:208)
[12:57:14] <klausman>	 docker pull docker-registry.wikimedia.org/golang1.21 # works fine
[12:57:33] <klausman>	 after that pull, I still get the same error as before
[12:57:40] <klausman>	 (when using build
[12:57:41] <klausman>	 )
[12:58:20] <klausman>	 if I use said image name in the FROM line (no {{}} at all), build works fine
[12:59:20] <klausman>	 I don't think it's a local setup problem, as testing the making of that earlier golang1.21, everything was working
[12:59:49] <klausman>	 again: I suspect whatever {{ .... | image_tag }} does breaks because there is a dot in the image name
[13:01:23] <chrisalbon>	 Friday!
[13:02:01] <klausman>	 Correct :)
[13:18:47] <elukey>	 klausman: can you send me a diff or file a patch so I can test?
[13:28:24] <klausman>	 I'll put it on Gerrit, is that ok?
[13:30:18] <elukey>	 yep
[13:31:34] <klausman>	 https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/967451
[13:35:33] <elukey>	 klausman: I commented, I think you are missing a -1 in the changelog
[13:36:21] <klausman>	 But that sure won't break building the image?
[13:36:42] <elukey>	 it is not a valid debian version afaics
[13:37:35] <klausman>	 I'm not saying it doesn't need to be fixed, but adding it also doesn't make the build work
[13:38:03] <elukey>	 sure sure
[13:38:27] <elukey>	 but the dot in the name shouldn't count, we already build with it golang1.18
[13:38:51] <klausman>	 Well, that didn't work, either, and not becaus ekserve needs 1.20+
[13:39:03] <klausman>	 maybe this broke since the last time we built kserve images
[13:40:43] <elukey>	 ah ok I found the issue
[13:40:48] <elukey>	 golang1.18 (1.18-1) wikimedia; urgency=high
[13:40:49] <elukey>	 vs
[13:40:54] <elukey>	 golang1.21 (1.21) wikimedia; urgency=high
[13:41:03] <elukey>	 so the issue is the 1.21  tag
[13:41:45] <klausman>	 oh, so the missing -1 on the golang image is the problem. Gotcha, sending a patch
[13:42:06] <elukey>	 I think so, but since you already published it we need a new entry in the changelog
[13:43:42] <elukey>	 klausman: you can try to build locally, hopefully it will work
[13:46:04] <wikibugs>	 10Machine-Learning-Team: Refactor LLM class and model server to run locally - https://phabricator.wikimedia.org/T349371 (10isarantopoulos)
[13:46:05] <klausman>	 REPOSITORY                                        TAG       IMAGE ID       CREATED             SIZE
[13:46:08] <klausman>	 docker-registry.wikimedia.org/golang1.21          1.21-1    acbcd2305163   35 seconds ago      621MB
[13:46:18] <klausman>	 ^^^ I now have that, but building kserve still fails with the same error
[13:46:20] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: llm: refactor to run locally [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/967455 (https://phabricator.wikimedia.org/T349371)
[13:46:36] <elukey>	 okok but we need to fix it anyway
[13:46:50] <elukey>	 so let's start with that code change
[13:47:13] <isaranto>	 o/ doint pay attention to the above patch it is WIP for now. I'll put it as active and add reviewers when it is tested
[13:47:20] <isaranto>	 *don't
[13:47:51] <klausman>	 elukey: https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/967456
[13:49:09] <elukey>	 ok perfect
[13:49:55] <klausman>	 it's only the tag in (), right, nothing else?
[13:50:58] <elukey>	 in theory yes
[13:51:08] <klausman>	 ok, merged and building now
[13:52:24] <elukey>	 I left a comment saying "wait for Janis approval first"
[13:52:30] <klausman>	 but he did
[13:52:43] <elukey>	 super didn't see
[13:52:44] <elukey>	 my bad
[13:53:46] <klausman>	 Ehi, è venerdì.
[13:55:51] <klausman>	 Still the same error when trying to build kserve.
[13:55:54] <elukey>	 the kserve build seems working now
[13:56:03] <elukey>	 at least for me locally
[13:56:13] <klausman>	 Still broken locally for me
[13:56:22] <elukey>	 try to remove all golang local images
[13:56:27] <elukey>	 1.21 I mean
[13:57:19] <klausman>	 yeah, done that, still no joy
[13:57:49] <elukey>	 it is working for me, I can see it building
[13:57:57] <elukey>	 * Built image docker-registry.wikimedia.org/kserve-build:0.11.1-1
[13:58:45] <elukey>	 klausman: what do you have in `docker image ls | grep golang` ?
[13:58:59] <klausman>	 Nope.
[13:59:06] <klausman>	 er, nothing
[13:59:28] <elukey>	 and if you run `docker-pkg build images` do you get the same error?
[14:00:07] <klausman>	 Running that now
[14:01:04] <klausman>	 Now it seems to work. Bloody voodo.
[14:01:08] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team, 10I18n, 10NewFunctionality-Worktype, 10Patch-For-Review: Create a language detection service in LiftWing - https://phabricator.wikimedia.org/T340507 (10isarantopoulos) Load testing was done using input text ranging from 5 to 500 words. Although 1-2 sentences would be...
[14:04:19] <elukey>	 klausman: if you try to run build for a specific directory is different that on the images dire as a whole
[14:04:29] <elukey>	 non bloody voodo :)
[14:04:47] <klausman>	 It really shouldn't be different, or at least emit some warning
[14:05:53] <wikibugs>	 10Machine-Learning-Team: [CI] Update pre-commit versions in inf-services repo - https://phabricator.wikimedia.org/T349382 (10isarantopoulos)
[14:06:00] <elukey>	 klausman: https://github.com/wikimedia/operations-docker-images-docker-pkg#build-the-images
[14:06:04] <wikibugs>	 10Machine-Learning-Team: [CI] Update pre-commit versions in inf-services repo - https://phabricator.wikimedia.org/T349382 (10isarantopoulos)
[14:06:09] <elukey>	 it looks for the image tags in the various changelogs
[14:07:07] <klausman>	 But the golang1.21 build worked just fine that way
[14:07:22] <klausman>	 maybe ebcause it was entirely new, not an update?
[14:07:59] <elukey>	 with the base images is different, your only requirement in there was bookworm
[14:09:34] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: ci: update pre-commit hooks versions [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/967458 (https://phabricator.wikimedia.org/T349382)
[14:15:28] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/967458 (https://phabricator.wikimedia.org/T349382) (owner: 10Ilias Sarantopoulos)
[14:17:05] <isaranto>	 made an attempt to update pre-commit versions in inf services. Things seem smooth locally
[14:17:37] <isaranto>	 is there any hack in gerrit/blubber to trigger all test pipelines?
[14:18:14] <elukey>	 isaranto: mmm you can try to log in to integration.wikimedia.org and trigger one job manually
[14:18:29] <wikibugs>	 (03PS2) 10Ilias Sarantopoulos: ci: update pre-commit hooks versions [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/967458 (https://phabricator.wikimedia.org/T349382)
[14:19:49] <isaranto>	 although: local run formats and checks all the repo so it would be fine
[14:20:01] * klausman going for an eagle stomp, bbiab
[14:44:30] <wikibugs>	 (03CR) 10Elukey: [C: 03+1] ci: update pre-commit hooks versions [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/967458 (https://phabricator.wikimedia.org/T349382) (owner: 10Ilias Sarantopoulos)
[15:13:24] <isaranto>	 Going afk for the weekend folks. Continuing on Monday :). Have a great weekend all!
[15:20:11] <elukey>	 o/
[15:36:18] <elukey>	 going afk as well! o/
[15:38:00] <klausman>	 \o
[16:19:29] <wikibugs>	 10Machine-Learning-Team: Investigate recommendation-api-ng internal endpoint failure - https://phabricator.wikimedia.org/T347475 (10Isaac) thanks @kevinbazira ! That API call worked for me as well. A few notes: * When I add the `&pageviews` parameter, the `pageviews` values in the response all become `null` inst...
[16:27:51] <wikibugs>	 10Machine-Learning-Team, 10Section-Level-Image-Suggestions, 10Patch-For-Review, 10Structured-Data-Backlog (Current Work): [XL] Productionize section alignment model training - https://phabricator.wikimedia.org/T325316 (10mfossati) =====Weekly update===== The SEAL pipeline is currently running in production...
[16:59:31] <wikibugs>	 10Machine-Learning-Team, 10Data-Engineering, 10Wikimedia Enterprise, 10Data Engineering and Event Platform Team (Sprint 3), and 2 others: [Event Platform] Implement PoC Event-Driven Data Pipeline for Revert Risk Model Scores using Event Platform Capabilities - https://phabricator.wikimedia.org/T338792 (10Ah...
[21:33:12] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Chinese-Sites, 10Growth-Team (Sprint 1 (Growth Team)), 10User-notice: Deploy "add a link" to 14th round of wikis - https://phabricator.wikimedia.org/T308139 (10Etonkovidova) 05Open→03Resolved