[05:26:50] <wikibugs>	 10Machine-Learning-Team: Deploy nllb-200 to production - https://phabricator.wikimedia.org/T349163 (10isarantopoulos)
[06:25:01] <isaranto>	 elukey: rec api is using default blubber python version 3.9
[06:26:30] <isaranto>	 so we don't need to change anything. If however they are easy fixes we can do that now to be futureproof. I also suggested running pytest without supressing the deprecation warnings (I dont see a reason why to do that as it can only lead to future errors)
[06:52:53] <wikibugs>	 10Machine-Learning-Team, 10Observability-Alerting, 10Patch-For-Review: Lift Wing alerting - https://phabricator.wikimedia.org/T346151 (10isarantopoulos) For the kafka lag when I try the query `kafka_burrow_partition_lag{           group="cpjobqueue-ORESFetchScoreJob"}` in thanos I see 2 topics  for codfw (on...
[07:00:10] <isaranto>	 Afk bbl
[07:02:55] <elukey>	 morning!
[07:03:12] <elukey>	 isaranto: yes yes I meant to add easy fixes if it means supporting python 3.11/bookworm
[07:20:28] <elukey>	 so the import error with python 3.11 comes from flaskrestplus IIUC
[07:20:29] <elukey>	 https://github.com/noirbizarre/flask-restplus/pull/768
[07:20:32] <elukey>	 2020
[07:23:10] <elukey>	 weird though, the code should work for our use case
[07:29:50] <elukey>	 the code does https://github.com/noirbizarre/flask-restplus/blob/master/flask_restplus/model.py#L9
[07:30:05] <elukey>	 that is correct, but I see another one
[07:30:07] <elukey>	 mmmm
[07:30:14] <elukey>	 maybe they committed without releasing
[07:30:40] <elukey>	 https://github.com/noirbizarre/flask-restplus/commit/136da9ddfa19b8eb30004553a017e84364c9ffd8
[07:31:07] <elukey>	 yeah
[07:32:00] <elukey>	 the project seems totally abandoned
[07:42:01] <elukey>	 anyway, forcing python3.9 in tox
[07:43:56] <elukey>	 but on Debian Bookworm it misses distutils, so tests are not running locally
[07:44:57] <elukey>	 need to use a container with bullseye then
[07:47:33] <kevinbazira>	 thanks for the review elukey. now going to test on staging...
[07:47:54] <elukey>	 ack super
[07:47:58] <elukey>	 let us know how it goes
[08:03:09] <kevinbazira>	 deployment completed, going to query the api ...
[08:08:13] <kevinbazira>	 running into this error:
[08:08:13] <kevinbazira>	 ```
[08:08:13] <kevinbazira>	 upstream connect error or disconnect/reset before headers. reset reason: connection termination
[08:08:13] <kevinbazira>	 ```
[08:12:20] <elukey>	 that is the external error, there must be something not working in the pod
[08:12:24] <kevinbazira>	 looked at the container logs and they have:
[08:12:24] <kevinbazira>	 ```
[08:12:24] <kevinbazira>	 404 Client Error: Not Found for url: http://localhost:6500/
[08:12:24] <kevinbazira>	 ```
[08:12:24] <kevinbazira>	 not sure whether these requests are going through the envoy proxy
[08:12:45] <elukey>	 good point. How to verify?
[08:15:33] <kevinbazira>	 I can't get into the container on staging, not sure how I can verify this
[08:16:19] <elukey>	 we can verify the settings
[08:16:50] <elukey>	 for example, the 404 that you are getting is related to a specific endpoint
[08:17:08] <elukey>	 and we have set localhost:6500, namely the mediawiki api
[08:17:37] <elukey>	 is the right endpoint?
[08:20:02] <elukey>	 namely what we set in https://gerrit.wikimedia.org/r/c/research/recommendation-api/+/965142/9/recommendation/data/recommendation_liftwing.ini
[08:20:40] <kevinbazira>	 yep, the languagepairs endpoint is failing and it uses cxserver.wikimedia.org
[08:21:04] <elukey>	 it uses localhost:6500 at the moment
[08:21:34] <elukey>	 (in [endpoints])
[08:22:13] <kevinbazira>	 ryep, pushing a fix for this
[08:25:32] <kevinbazira>	 it actually uses the right host header: https://github.com/wikimedia/research-recommendation-api/blob/5e138ba4a0d5d448deca3d178d078c23ff4752ac/recommendation/data/recommendation_liftwing.ini#L10C1-L10C40
[08:26:03] <kevinbazira>	 this should work unless wikimedia.org uses a different listener other than 6500
[08:27:37] <elukey>	 is cxserver.wikimedia.org served by the wikimedia php api?
[08:28:09] <elukey>	 (same question for intake-analytics.wikimedia.org)
[08:31:20] <kevinbazira>	 the mw-api-int-async-ro envoy listener is only for wikimedia php api?
[08:31:30] <elukey>	 yep
[08:32:32] <elukey>	 we need to add the listeners that we need, and set the right localhost:port combination
[08:32:34] <kevinbazira>	 by any chance have you come across documentation that details this?
[08:33:02] <kevinbazira>	 looking at the envoy listeners listed here doesnt clarify that: https://gerrit.wikimedia.org/g/operations/puppet/+/refs/heads/production/hieradata/common/profile/services_proxy/envoy.yaml
[08:33:51] <kevinbazira>	 it doesn't clarify mw-api-int-async-ro envoy listener is only for wikimedia php api 
[08:34:08] <kevinbazira>	 this what caused my confusion
[08:34:22] <elukey>	 sure sure it can happen
[08:34:46] <elukey>	 there are separate entries for other endpoints, like cxserver etc..
[08:35:09] <elukey>	 they are separate services, usually when people talk about the mw-api they assume the PHP one
[08:35:27] <elukey>	 (at least in my experience)
[08:36:41] <kevinbazira>	 sure, so I see cxserver uses 6015
[08:39:44] <elukey>	 intake-analytics.wikimedia.org is served by an eventgate instance
[08:39:54] <elukey>	 should be port 6004
[08:40:11] <elukey>	 but we need to add both listeners to the mesh config in deployment-charts
[08:42:49] <kevinbazira>	 sure sure
[08:56:08] <kevinbazira>	 interesting, this example shows that the 6500 port (the same as mw-api-int-async-ro) is also used for wikipedia.org: https://wikitech.wikimedia.org/wiki/Envoy#Example_(calling_mw-api)
[09:02:15] <elukey>	 kevinbazira: yeah it makes sense, it is the mw php api
[09:02:19] <elukey>	 what is your doubt?
[09:05:12] <kevinbazira>	 we have got the following so far:
[09:05:12] <kevinbazira>	 language_pairs = cxserver.wikimedia.org: cxserver - 6015
[09:05:12] <kevinbazira>	 pageviews = wikimedia.org: mw-api-int-async-ro - 6500
[09:05:12] <kevinbazira>	 wikipedia = {source}.wikipedia.org: mw-api-int-async-ro - 6500
[09:05:12] <kevinbazira>	 wikidata = www.wikidata.org: mw-api-int-async-ro - 6500
[09:05:12] <kevinbazira>	 event_logger = intake-analytics.wikimedia.org: eventgate-analytics - 6004
[09:05:13] <kevinbazira>	 one left:
[09:05:26] <kevinbazira>	 related_articles = recommend-related-articles.wmflabs.org: ???
[09:08:09] <elukey>	 it seems another api, not sure if it is used
[09:08:26] <elukey>	 candidate_finders.py seems to use it
[09:16:20] <elukey>	 kevinbazira: I tried with curl, it seems a dead end
[09:16:29] <elukey>	 it points to the wmflabs proxy, but nothing is configured
[09:16:34] <elukey>	 also checked in https://openstack-browser.toolforge.org/proxy/
[09:27:59] <wikibugs>	 (03PS2) 10Elukey: Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542
[09:28:01] <wikibugs>	 (03PS2) 10Elukey: Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543
[09:29:36] <wikibugs>	 (03CR) 10Elukey: Add precommit support (032 comments) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 (owner: 10Elukey)
[09:31:16] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: Add precommit support (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 (owner: 10Elukey)
[09:32:31] <kevinbazira>	 yep elukey I looked at the full url as configured by the original devs and it looks like it's down:
[09:32:31] <kevinbazira>	 https://github.com/wikimedia/research-recommendation-api/blob/5e138ba4a0d5d448deca3d178d078c23ff4752ac/recommendation/data/recommendation.ini#L7C20-L7C101
[09:32:31] <kevinbazira>	 changed it to point to the current host on wmflabs and still it's down:
[09:32:31] <kevinbazira>	 https://recommend.wmflabs.org/types/related_articles/v1/articles/
[09:32:31] <kevinbazira>	 in T339890#9179995 Isaac advised that we switch off related_articles, so is likely not to affect the LiftWing instance
[09:32:31] <kevinbazira>	 I am going to push changes for the listeners we listed above
[09:34:12] <elukey>	 ack
[09:35:56] <elukey>	 isaranto: one qs - should I just use "tox" as entrypoint in blubber?
[09:37:49] <isaranto>	 elukey: yes - if you move the command under testenv
[09:38:00] <elukey>	 yep yep it is a good point
[09:38:05] <isaranto>	 tox.ini would be sth like this 
[09:38:12] <isaranto>	 https://www.irccloud.com/pastebin/3qTHdf5E/
[09:38:22] <elukey>	 exactly yes
[09:38:29] <elukey>	 and we shouldn't need a dedicated entrypoint.sh
[09:38:41] <elukey>	 I see that in inference-services we initialize git
[09:39:25] <isaranto>	 a w8. we would need to do that probably.
[09:39:30] <elukey>	 if needed I can change run-tests.sh in the rec-api repo like we do for inference-services
[09:40:06] <isaranto>	 pre-commit runs for a git repo and they we handle ci is different. we just copy files in a container and don't checkout the current repo/commit
[09:40:33] <isaranto>	 when we move to gitlab iiuc we could change this behavior
[09:41:26] <wikibugs>	 (03PS3) 10Elukey: Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542
[09:41:28] <wikibugs>	 (03PS3) 10Elukey: Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543
[09:41:54] <wikibugs>	 (03CR) 10Elukey: Add precommit support (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 (owner: 10Elukey)
[09:42:01] <wikibugs>	 (03CR) 10Elukey: Add precommit support (031 comment) [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 (owner: 10Elukey)
[09:42:51] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 (owner: 10Elukey)
[09:43:14] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey)
[09:43:26] <elukey>	 ./run-test.sh: line 2: git: command not found
[09:43:29] <elukey>	 right :D
[09:45:32] <elukey>	 testing locally this time to see if it works
[09:46:25] <wikibugs>	 (03PS1) 10Kevin Bazira: Update external endpoint ports used on LiftWing [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966826 (https://phabricator.wikimedia.org/T348607)
[09:46:51] <wikibugs>	 (03CR) 10Elukey: [C: 03+1] Update external endpoint ports used on LiftWing [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966826 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira)
[09:47:33] <kevinbazira>	 woah, thanks elukey. that was fast! :)
[09:47:59] <wikibugs>	 (03CR) 10Kevin Bazira: [C: 03+2] Update external endpoint ports used on LiftWing [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966826 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira)
[09:48:13] <elukey>	 it is a quick one, we already discussed the changes above :)
[09:48:31] <wikibugs>	 (03Merged) 10jenkins-bot: Update external endpoint ports used on LiftWing [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966826 (https://phabricator.wikimedia.org/T348607) (owner: 10Kevin Bazira)
[09:50:55] <wikibugs>	 (03PS4) 10Elukey: Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542
[09:50:57] <wikibugs>	 (03PS4) 10Elukey: Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543
[09:52:03] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 (owner: 10Elukey)
[09:52:32] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey)
[09:53:09] <elukey>	 I wanted to see if it was only local, but I see
[09:53:10] <elukey>	 AttributeError: module 'virtualenv.create.via_global_ref.builtin.cpython.mac_os' has no attribute 'CPython3macOsBrew'
[09:53:48] <isaranto>	 aa w8 I witnessed this issue in another patch recently
[09:54:35] <elukey>	 I recall something weird similar
[09:54:51] <elukey>	 maybe the virtualenv version or something?
[09:55:08] <isaranto>	 yes virtualenv incompatibility between libraries
[09:55:28] <isaranto>	 a found it
[09:55:32] <elukey>	 we pin virtualenv==20.23.1
[09:55:49] <elukey>	 and there is also https://github.com/tox-dev/tox/pull/1537
[09:55:52] <isaranto>	 in the langid image.
[09:55:52] <isaranto>	 I added `pip uninstall virtualenv -y` in the entyrpoint.sh script
[09:56:21] <isaranto>	 so that we allow tox to do its thing
[09:56:24] <elukey>	 yes now I recall, the "please don't ask" :D
[09:57:02] <elukey>	 but what if I use a different virtualenv?
[09:57:29] <elukey>	 trying to build
[09:59:01] <isaranto>	 perhaps that could work too (using a different virtualenv
[09:59:07] * isaranto afk early lunch!
[09:59:08] <elukey>	 not pinning works, but then I see ERROR: No matching distribution found for flask==1.1.4
[09:59:56] <elukey>	 and it seems that that version doesn't support py39
[10:24:12] <elukey>	 it is all a mess with the debian image I think, trying to tweak some settings
[10:34:21] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team: Discuss caching strategies for Lift Wing - https://phabricator.wikimedia.org/T349180 (10klausman)
[10:35:02] <wikibugs>	 10Machine-Learning-Team, 10Goal: Goal: Decide on an optional Lift Wing caching strategy for model servers - https://phabricator.wikimedia.org/T348155 (10klausman)
[10:35:04] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team: Discuss caching strategies for Lift Wing - https://phabricator.wikimedia.org/T349180 (10klausman)
[10:44:33] * elukey lunch!
[10:44:39] <klausman>	 ditto
[10:46:55] <wikibugs>	 10Machine-Learning-Team: Decommission ORES configurations and servers - https://phabricator.wikimedia.org/T347278 (10jijiki)
[10:48:05] <wikibugs>	 10Machine-Learning-Team: Decommission ORES configurations and servers - https://phabricator.wikimedia.org/T347278 (10jijiki) Data has been flushed from both rdb1011 and rdb2009
[11:07:15] <isaranto>	 lemm know if I can help with the python images for rec-api
[12:14:11] <elukey>	 isaranto: hopefuly I should be able to fix it in a bit, very weird
[12:58:09] <chrisalbon>	 Morning
[12:58:46] <elukey>	 morning!
[12:59:06] <isaranto>	 o/
[13:49:37] <wikibugs>	 (03PS5) 10Elukey: Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542
[13:49:39] <wikibugs>	 (03PS5) 10Elukey: Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543
[13:49:49] <elukey>	 testing to see if another problem is only local/mine --^
[13:50:35] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Add precommit support [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966542 (owner: 10Elukey)
[13:51:14] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Fix pre-commit errors and bump version [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/966543 (owner: 10Elukey)
[13:52:06] <elukey>	 without virtualenv I get
[13:52:07] <elukey>	 ERROR: Could not find a version that satisfies the requirement flask==1.1.4 (from versions: none)
[13:53:22] <elukey>	 it is definitely an issue between what debian installs and what pip installs
[13:53:30] <elukey>	 since with a regular debian:bullseye container all works
[14:30:10] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team: Discuss caching strategies for Lift Wing - https://phabricator.wikimedia.org/T349180 (10calbon)
[14:53:45] <isaranto>	 so flask 1.1.4 is really old. I wonder if it can be upgraded easily
[14:53:52] <isaranto>	 "easily" 😓
[14:57:53] <isaranto>	 I regret our decision not to move it to fastapi since we'll eventually need work to move upgrade it anyway
[15:00:10] <elukey>	 I tried to upgrade but it gives the same issue, it seems more an issue with virtualenvs/pip/etc..
[15:00:15] <elukey>	 I'll figure it out
[15:00:19] <elukey>	 there is no rush
[15:53:50] <isaranto>	 pff can't get the alert right
[15:54:02] <isaranto>	 starting fresh tomorrow!
[15:59:18] <isaranto>	 added a patch for nllb - https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/966891/
[15:59:47] <isaranto>	 logging off, have a nice evening/day folks!
[16:01:14] <elukey>	 you too
[16:07:12] <elukey>	 going afk too!
[16:07:19] <elukey>	 have a nice rest of the day folks
[21:17:09] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Sprint 1 (Growth Team)), 10User-notice: Deploy "add a link" to 15th round of wikis - https://phabricator.wikimedia.org/T308141 (10KStoller-WMF)