[05:49:42] <isaranto>	 Morning o/
[07:42:30] <wikibugs>	 10Machine-Learning-Team: Upgrade model servers to kserve 0.11.2 - https://phabricator.wikimedia.org/T351633 (10isarantopoulos) 05Open→03Resolved
[07:47:39] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: Enable local runs for article-descriptions model - https://phabricator.wikimedia.org/T351940 (10isarantopoulos) Currently investigating the following error I get on model load when setting low_cpu_usage=True : `  File "/Users/isaranto/repoz/inference-services/articl...
[08:01:29] <isaranto>	 Afk for 1h
[08:09:55] <wikibugs>	 (03CR) 10Kosta Harlan: Change default config values to support local/patchdemo deployments [extensions/ORES] - 10https://gerrit.wikimedia.org/r/976157 (https://phabricator.wikimedia.org/T351703) (owner: 10Ilias Sarantopoulos)
[08:09:59] <wikibugs>	 (03CR) 10Kosta Harlan: [C: 03+2] Change default config values to support local/patchdemo deployments [extensions/ORES] - 10https://gerrit.wikimedia.org/r/976157 (https://phabricator.wikimedia.org/T351703) (owner: 10Ilias Sarantopoulos)
[08:13:00] <wikibugs>	 (03Merged) 10jenkins-bot: Change default config values to support local/patchdemo deployments [extensions/ORES] - 10https://gerrit.wikimedia.org/r/976157 (https://phabricator.wikimedia.org/T351703) (owner: 10Ilias Sarantopoulos)
[09:03:37] <isaranto>	 Back!
[09:16:23] <elukey>	 morning!
[09:41:00] <wikibugs>	 10Machine-Learning-Team: Apply multi-processing to preprocess() in isvcs that suffer from high latency - https://phabricator.wikimedia.org/T349274 (10elukey) a:05elukey→03None
[09:50:53] <wikibugs>	 (03PS18) 10Ilias Sarantopoulos: add revertrisk model to the list of models [extensions/ORES] - 10https://gerrit.wikimedia.org/r/971547 (https://phabricator.wikimedia.org/T348298)
[10:36:24] <klausman>	 G'day
[10:36:39] <klausman>	 kevinbazira: will push the a-d update once Jenkins has merged it
[10:37:38] <kevinbazira>	 klausman: o/
[10:37:43] <kevinbazira>	 great. thanks!!
[10:40:53] <klausman>	 And it's running. Old pod still shutting down
[10:42:43] <klausman>	 Old pods are gone. I'll tail the logs to see if-when testing shows anything 
[10:46:11] <klausman>	 kevinbazira: it just threw a 500: https://phabricator.wikimedia.org/P53917
[10:46:55] <klausman>	 Also, it seems the wmapi is returning 404s
[10:47:11] <kevinbazira>	 klausman: thanks. I do not have permission to view the link you shared :)
[10:47:49] <klausman>	 should be visible now
[10:48:44] <klausman>	 https://phabricator.wikimedia.org/P53918 and here's the 404s
[10:50:22] <kevinbazira>	 ok, both 404 are because of the internal endpoint: `api-ro.discovery.wmnet`
[10:53:04] <wikibugs>	 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Growth-Team, 10Wikipedia-Android-App-Backlog, 10Patch-For-Review: Add revertrisk-language-agnostic to RecentChanges filters - https://phabricator.wikimedia.org/T348298 (10Samwalton9-WMF) We've come to realise that this task is important for both the...
[10:57:07] <klausman>	 So are the 404s expected in normal operation?
[11:02:10] <kevinbazira>	 no they are not expected
[11:02:47] <kevinbazira>	 is there a way you can confirm whether we can hit this endpoint on k8s/LiftWing?
[11:02:47] <kevinbazira>	 ```
[11:02:47] <kevinbazira>	 'http://api-ro.discovery.wmnet/api/rest_v1/page/summary/Clandonald'
[11:02:47] <kevinbazira>	 ```
[11:03:02] <klausman>	 I'll give it a shot
[11:03:56] <klausman>	 What is the Host: header to use?
[11:05:02] <klausman>	 For a direct request, I am getting an Unconfigured domain page
[11:10:23] <kevinbazira>	 the host header is `*.wikipedia.org` in this case we could test with: `en.wikipedia.org`
[11:11:46] <klausman>	 I'll c&p the HTML I get
[11:12:50] <klausman>	 https://phabricator.wikimedia.org/P53919
[11:14:19] <klausman>	 This ran from inside the kserve container inside of article-descriptions-predictor-default-00005-deployment-c6x4tqh
[11:14:42] <elukey>	 "Did you mean: " ;)
[11:16:48] <elukey>	 (inside the paste_
[11:16:50] <elukey>	 )
[11:17:03] <elukey>	 (I think a /wiki/ is missing)
[11:17:06] * elukey lunch
[11:17:41] <kevinbazira>	 great. so the "did you mean" part is suggesting to use: `/wiki/api/rest_v1/page/summary/Clandonald`
[11:17:58] <kevinbazira>	 instead of `api/rest_v1/page/summary/Clandonald`
[11:18:32] <kevinbazira>	 thank you for sharing the log klausman. let me push a patch.
[11:19:12] <klausman>	 I've tried using the suggest URL, but so far, that just hangs
[11:19:21] <klausman>	 i.e. >>> url = 'http://api-ro.discovery.wmnet/wiki/api/rest_v1/page/summary/Clandonald'
[11:19:35] <kevinbazira>	 o_o
[11:20:20] <klausman>	 Kinda wild.
[11:21:39] <klausman>	 ah, this works:
[11:21:47] <klausman>	 http://api-ro.discovery.wmnet/wiki/Api/rest_v1/page/summary/Clandonald
[11:21:57] <klausman>	 (I think. It's a lot of HTML)
[11:22:34] <kevinbazira>	 ok, we want it to return something similar to what this returns: https://en.wikipedia.org/api/rest_v1/page/summary/Clandonald
[11:23:46] <klausman>	 yeah, no, I get a whole HTML page, no JSON
[11:24:22] <klausman>	 I think the hint to go to wiki/api is a red herring, that is not supposed to happen
[11:28:44] <kevinbazira>	 klausman: how about we try: `http://api-ro.discovery.wmnet/v1/page/summary/Clandonald`
[11:29:01] <klausman>	 sec
[11:29:37] <klausman>	 nope, same "we could not find this page" messgae
[11:30:43] <kevinbazira>	 hmmm... I suggested that because in the rec-api we couldn't hit this endpoint on k8s:
[11:30:43] <kevinbazira>	 ```
[11:30:43] <kevinbazira>	 https://wikimedia.org/api/rest_v1/metrics/pageviews
[11:30:43] <kevinbazira>	 ```
[11:30:43] <kevinbazira>	 so elukey advised that we change it to:
[11:30:44] <kevinbazira>	 ```
[11:30:44] <kevinbazira>	 http://localhost:6033/wikimedia.org/v1/metrics/pageviews
[11:30:45] <kevinbazira>	 ```
[11:30:45] <kevinbazira>	 with the `wikimedia.org` set as the host header and it worked.
[11:32:00] <kevinbazira>	 so in this case we are trying to access:
[11:32:00] <kevinbazira>	 ```
[11:32:00] <kevinbazira>	 https://en.wikipedia.org/api/rest_v1/page/summary/Clandonald
[11:32:00] <kevinbazira>	 ```
[11:32:00] <kevinbazira>	 so far we have set `*.wikipedia.org` as the host header and are using `api-ro.discovery.wmnet` as the internal k8s endpoint but this endpoint is failing:
[11:32:01] <kevinbazira>	 ```
[11:32:01] <kevinbazira>	 http://api-ro.discovery.wmnet/api/rest_v1/page/summary/Clandonald
[11:32:02] <kevinbazira>	 ```
[11:32:02] <kevinbazira>	 we definitely need to adjust this path `api/rest_v1/page/summary/Clandonald`
[11:34:45] <klausman>	 oh hang on, I may have done a dumb
[11:37:32] <klausman>	 So I am trying http://api-ro.discovery.wmnet/api/v1/page/summary/Clandonald with Host: 'wikimedia.org', but that just gives me a 502
[11:37:50] <klausman>	 I have also tried H: en.wikipedia.org, but that gives the "we don't know this page" message
[11:38:19] <klausman>	 Same URL, no /api prefix, and the wm.org Host, also 502.
[11:39:03] <klausman>	 It's puzzling to me that H: en.... and the same URL-Path as external ("/api/....") doesn't just work.
[11:42:43] <kevinbazira>	 yep, we experienced this issue with the rec-api. endpoints internal to k8s don't seem to work the same way external ones do and there doesn't seem to be clear documentation on why that is :)
[11:49:57] <klausman>	 I also looked at the Istio logs, and it just logs this JSON event: https://phabricator.wikimedia.org/P53921
[11:50:22] <klausman>	 Which seems like it's just doing what we're asking it to
[11:51:23] <klausman>	 elukey: I am out of ideas here. Would appreciate a hint :)
[11:51:26] <taavi>	 to me it seems like you're trying to send some requests to mediawiki even though the endpoints are handled by some other service
[11:51:43] <taavi>	 /api/rest_v1/ for example is handled by restbase per https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/profile/trafficserver/backend.yaml#309
[11:54:00] <wikibugs>	 10Machine-Learning-Team, 10GrowthExperiments, 10Growth-Team (Sprint 3 (Growth Team)), 10MW-1.42-notes (1.42.0-wmf.7; 2023-11-28): importOresTopics script fails to import topics - https://phabricator.wikimedia.org/T350137 (10Urbanecm_WMF) 05Open→03Resolved
[11:57:17] <klausman>	 With H: restbase and the original URL (api-ro), I get a 502, but I don't know *yet) from what
[11:59:30] <klausman>	 Probably Istio itself, since we haven't configured it in ml-serve.yaml
[12:01:23] <klausman>	 I'll have a think on this over lunch. bbiab
[12:14:13] <chrisalbon>	 Morning all
[12:27:15] <isaranto>	 o/
[12:27:28] * isaranto afk lunch
[13:08:11] <chrisalbon>	 wikibugs:
[13:24:13] <isaranto>	 🐛
[13:31:14] <elukey>	 hey folks
[13:31:22] <elukey>	 back :)
[13:32:26] <elukey>	 klausman: what taa*vi suggested is that we hit the restbase discovery endpoint
[13:32:35] <elukey>	 something that we don't currently allow in the mesh
[13:33:07] <elukey>	 kevinbazira: re: localhost:port, that combination is only needed for services outside the mesh, not isvcs (so ores-legacy and rec-api-ng for the moment)
[13:33:13] <elukey>	 we discussed it during the last hack meeting
[13:35:36] <elukey>	 so, the external URL for restbase is https://en.wikipedia.org/api/rest_v1/page/summary/Clandonald
[13:35:39] <elukey>	 and that works
[13:36:09] <elukey>	 but as taa*vi mentioned, it doesn't hit (after the CDN) api-ro.discovery.wmnet, but Restbase, that has a different endpoint
[13:37:50] <elukey>	 just to add more difficulty, RestBase is currently being deprecated in favor of rest-gateway.discovery.wmnet
[13:37:55] <elukey>	 that has the same API though
[13:38:14] <elukey>	 so in service.yaml (puppet), rest-gateway.discovery.wmnet seems to listen on port 4113
[13:42:30] <elukey>	 kevinbazira: I don't recall what we did for rec-api-ng, what URI do we use?
[13:51:10] <elukey>	 mmm if I try the following from stat1004 it hangs
[13:51:11] <elukey>	 https://rest-gateway.discovery.wmnet:4113/wikimedia.org/v1/page/summary/Clandonald
[13:57:23] <kevinbazira>	 elukey: in the rec-api-ng the internal endpoints and host headers we used are:
[13:57:23] <kevinbazira>	 https://github.com/wikimedia/research-recommendation-api/blob/master/recommendation/data/recommendation_liftwing.ini#L1-L15
[13:57:23] <kevinbazira>	 we used them to access these external endpoints:
[13:57:23] <kevinbazira>	 https://github.com/wikimedia/research-recommendation-api/blob/master/recommendation/data/recommendation.ini#L1-L7
[13:58:35] <elukey>	 yep yep, so we used the rest gateway in that case
[13:58:43] <elukey>	 but it may not be fully compatible with restbase yet
[13:58:55] <elukey>	 trying to make the restbase.discovery.wmnet URL to work
[13:59:31] <kevinbazira>	 okok
[14:05:36] <elukey>	 ahhh okok 
[14:05:38] <elukey>	 https://restbase.discovery.wmnet:7443/en.wikipedia.org/v1/page/summary/Clandonald
[14:05:41] <elukey>	 this is the right one
[14:05:51] <elukey>	 let's see if rest-gateway is similar
[14:06:20] <elukey>	 yeah https://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/page/summary/Clandonald works
[14:06:29] <elukey>	 all right I think I have it
[14:06:55] <elukey>	 so, to recap:
[14:07:32] <elukey>	 a request like the following, https://en.wikipedia.org/api/rest_v1/page/summary/Clandonald
[14:07:55] <elukey>	 hits our external CDN cache layer (haproxy + varnish + ATS, if you recall from the hack time)
[14:09:02] <elukey>	 in there, as taa*vi mentioned, there is a rule that eventually (since we are in the middle of a migration) hits rest-gateway.discovery.wmnet:4113
[14:09:32] <elukey>	 this proxy config is stored in the CDN config on puppet, like https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/profile/trafficserver/backend.yaml#309
[14:09:56] <elukey>	 at the moment you see restbase.discovery.wmnet:7443 but it will be deprecated soon, so we use the new one directly 
[14:10:30] <elukey>	 the discovery endpoints that we just mentioned are not api-ro.discovery.wmnet, that is the MW API load balanced vip
[14:10:50] <elukey>	 Restbase is https://wikitech.wikimedia.org/wiki/RESTBase, if you are curious
[14:11:06] <elukey>	 behind the scenes it calls api-ro.discovery.wmnet too, but the APIs are different
[14:11:21] <elukey>	 so, the proxy rule is the following:
[14:11:37] <elukey>	 https://en.wikipedia.org/api/rest_v1/page/summary/Clandonald becomes
[14:11:48] <elukey>	 https://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/page/summary/Clandonald
[14:12:02] <elukey>	 without a specific Host header should work fine
[14:12:16] <elukey>	 note how en.wikipedia.org is inserted directly in the URI
[14:12:37] <elukey>	 there are two tasks to be made:
[14:13:05] <elukey>	 1) Allow the Istio mesh to contact rest-gateway.discovery.wmnet (the three configs in ml-serve.yaml in deployment-charts)
[14:13:42] <elukey>	 2) the isvc code needs to be adapted to call the rest-gateway directly (where /api/rest_v1 is used)
[14:13:49] <elukey>	 3) thanks taavi for the help :)
[14:14:12] <elukey>	 kevinbazira, klausman: lemme know if anything is not clear --^
[14:16:18] <kevinbazira>	 thank you for the explanation elukey. in the model-server we were using:
[14:16:19] <kevinbazira>	 ```
[14:16:19] <kevinbazira>	 http://api-ro.discovery.wmnet/api/rest_v1/page/summary/Clandonald
[14:16:19] <kevinbazira>	 ```
[14:16:19] <kevinbazira>	 I am going to push a patch that updates it to:
[14:16:19] <kevinbazira>	 ```
[14:16:19] <kevinbazira>	 https://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/page/summary/Clandonald
[14:16:20] <kevinbazira>	 ```
[14:21:22] <elukey>	 kevinbazira: I'd wait for Tobias, since we need to add the mesh config before that. If you recall during the hack time we went through how we configure the MW API and eventgate, using HTTP vs HTTPS ports to allow metrics. In this case we need to create another HTTP port (say 4110) and use http:// in the isvc code
[14:22:15] <kevinbazira>	 sure sure. I'll prepare the patch and wait to merge. is that ok?
[14:23:12] <elukey>	 we need to decide the http port first, Tobias will do it in deployment-charts, after that you'll have the final URL to call 
[14:23:57] <kevinbazira>	 okok
[14:36:13] <elukey>	 but please let me know if anything is unclear
[14:36:24] <elukey>	 we can have a chat over meet if you prefer
[14:36:30] <elukey>	 (localhost:port vs mesh etc..)
[14:36:42] <klausman>	 elukey: sounds good re: proxy/mesh changes
[14:37:40] <elukey>	 ack, are you going to take care of the patch?
[14:37:45] <elukey>	 I can review it when ready
[14:37:47] <klausman>	 yes
[14:38:05] <klausman>	 currently sorting a messed up git checkout
[14:41:56] <klausman>	 elukey: should we just c&p the series of hosts from the api-ro VS to the restgw VS? Or just use wikipedia (base and wildcard)?
[14:43:47] <elukey>	 klausman: I think the wikipedia domains should be enough, not sure if the other works
[14:43:55] <elukey>	 we should test and see
[14:44:03] <klausman>	 ack. we can always add them later
[14:44:05] <elukey>	 I think that maybe wikidata works, but not the others
[14:49:59] <klausman>	 elukey: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/978059 --- Not sure I got the ports (4110/4113) right
[14:51:46] <wikibugs>	 10Machine-Learning-Team: refactor revertrisk model server to run locally - https://phabricator.wikimedia.org/T352181 (10isarantopoulos)
[14:52:35] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: Enable local runs for article-descriptions model - https://phabricator.wikimedia.org/T351940 (10isarantopoulos)
[14:52:45] <wikibugs>	 10Machine-Learning-Team: refactor revertrisk model server to run locally - https://phabricator.wikimedia.org/T352181 (10isarantopoulos)
[14:55:23] <elukey>	 klausman: I came up with 4110 without checking, if it is used in puppet by another service definition we may want something different
[14:55:34] <elukey>	 (so in case we'll need to add that service, nothing will clash)
[14:56:18] <klausman>	 4110 is already used I think. hieradata/common/service.yaml mentions it in "Similar-users" (sockpuppet detection)
[14:56:33] <elukey>	 yeah so let's use another one
[14:56:44] <klausman>	 4111 is unsued, I think, let's use that
[14:57:07] <elukey>	 sure
[14:57:23] <wikibugs>	 10Machine-Learning-Team: Deploy ctranslate2 version of nllb-200 - https://phabricator.wikimedia.org/T351740 (10isarantopoulos)
[14:57:25] <klausman>	 change has been updated
[14:58:10] <elukey>	 klausman: looks good, maybe let's add some info to the description and Bug: #task so we have a reference
[14:59:01] <wikibugs>	 10Machine-Learning-Team: Document load test results - https://phabricator.wikimedia.org/T351939 (10isarantopoulos)
[14:59:40] <klausman>	 updated
[15:00:33] <wikibugs>	 10Machine-Learning-Team: Bump model servers to debian Bookworm - https://phabricator.wikimedia.org/T352184 (10isarantopoulos)
[15:40:12] <wikibugs>	 10Machine-Learning-Team: Document procedure for updating the underlying distro for model servers (e.g. Bullseye -> Bookworm) - https://phabricator.wikimedia.org/T352189 (10klausman)
[15:41:20] <wikibugs>	 10Machine-Learning-Team, 10Goal: Goal: Increase the number of models hosted on Lift Wing - https://phabricator.wikimedia.org/T348156 (10kevinbazira) Working on migrating the machine-generated article descriptions model from toolforge to LiftWing: - added article-descriptions model-server to LiftWing inference...
[15:47:15] <wikibugs>	 10Machine-Learning-Team: refactor revertrisk model server to run locally - https://phabricator.wikimedia.org/T352181 (10isarantopoulos) p:05Triage→03Medium
[15:48:35] <wikibugs>	 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Growth-Team, 10Wikipedia-Android-App-Backlog, 10Patch-For-Review: Add revertrisk-language-agnostic to RecentChanges filters - https://phabricator.wikimedia.org/T348298 (10isarantopoulos) p:05Triage→03Medium
[15:48:57] <wikibugs>	 10Machine-Learning-Team: Document procedure for updating the underlying distro for model servers (e.g. Bullseye -> Bookworm) - https://phabricator.wikimedia.org/T352189 (10calbon) p:05Triage→03Medium
[15:50:31] <wikibugs>	 10Machine-Learning-Team: Bump model servers to debian Bookworm - https://phabricator.wikimedia.org/T352184 (10calbon) a:03isarantopoulos
[15:51:50] <wikibugs>	 10Machine-Learning-Team, 10Product-Analytics, 10User-Iflorez: Transient error while running lift wing topic model - https://phabricator.wikimedia.org/T351114 (10calbon) p:05Triage→03Medium
[15:51:57] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: Enable local runs for article-descriptions model - https://phabricator.wikimedia.org/T351940 (10isarantopoulos) a:03isarantopoulos
[15:52:18] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: Enable local runs for article-descriptions model - https://phabricator.wikimedia.org/T351940 (10calbon) p:05Triage→03Medium
[15:52:46] <wikibugs>	 10Machine-Learning-Team: Document load test results - https://phabricator.wikimedia.org/T351939 (10calbon) p:05Triage→03Medium
[15:55:52] <wikibugs>	 10Machine-Learning-Team, 10serviceops, 10Patch-For-Review: Bump istio Docker images to Bookworm - https://phabricator.wikimedia.org/T351933 (10calbon) p:05Triage→03Medium
[15:56:29] <wikibugs>	 10Machine-Learning-Team: Document procedure for updating the underlying distro for model servers (e.g. Bullseye -> Bookworm) - https://phabricator.wikimedia.org/T352189 (10isarantopoulos)
[15:56:31] <wikibugs>	 10Machine-Learning-Team: Document load test results - https://phabricator.wikimedia.org/T351939 (10isarantopoulos)
[16:20:48] <klausman>	 elukey: the istio change should just be a `helmfile -e ml-staging-codfw apply` from admin_ng, right?
[16:22:25] <elukey>	 yep
[16:22:53] <klausman>	 ok, pushing,
[16:27:47] <klausman>	 SO we assume now that a plain request (No Host-header) to https://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/page/summary/Clandonald should just work, right?
[16:28:37] <klausman>	 Because I am still getting timeouts from inside the container
[16:28:55] <klausman>	 Oh wai....
[16:29:20] <elukey>	 not https
[16:29:41] <klausman>	 ah, right
[16:30:37] <klausman>	 http:// to :4113 gives me a conn refused, to :4111 I get a timeout
[16:32:19] <elukey>	 try to kill/restart the pod just in case, if the envoy proxy needs an encouragement
[16:32:25] <klausman>	 ack
[16:32:25] <elukey>	 in theory no but..
[16:35:27] <klausman>	 same deal (timeout/conn refused)
[16:36:21] <klausman>	 ah, I think I know what the problem is
[16:37:10] <klausman>	 so in the example of event gate http-pott is num 4480, target 4492, and then 4492 as the https port (service entry), and in the dDR, it uses 4480 again
[16:37:34] <klausman>	 But my config (4111 -> 4113) uses 4113 in the DR, which is wrong
[16:39:19] <klausman>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/978098 when you have a moment
[16:56:50] <elukey>	 klausman: +1, maybe just do a quick kubectl edit to the dr to verify?
[16:56:58] <klausman>	 good call
[16:56:59] <elukey>	 in case there is anything more etc..
[16:57:55] <klausman>	 what NS is that in?
[16:59:50] <klausman>	 elukey: or rather, what object would I be editing. It's not just in the exp namespace, so istio-system, I guess?
[17:01:25] <elukey>	 klausman: should be in knative-serving IIRC
[17:07:38] <klausman>	 That wssn't it, still getting timeout :-(
[17:08:45] <elukey>	 mmm weird, what is the URL that you are checking? (to double check)
[17:09:15] <klausman>	 http://rest-gateway.discovery.wmnet:4111/en.wikipedia.org/v1/page/summary/Clandonald
[17:09:42] <klausman>	 basically what kevin mentioned earlier, but s/https/http/
[17:13:03] <klausman>	 I run `import requests;r=requests.get('http://rest-gateway.discovery.wmnet:4111/en.wikipedia.org/v1/page/summary/Clandonald', timeout=10)` and it always runs into the 10s timeout.
[17:20:02] <isaranto>	 going afk folks, cu tomorrow
[17:24:30] <klausman>	 Also note:
[17:24:31] <klausman>	 $ curl http://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/page/summary/Clandonald
[17:24:33] <klausman>	 curl: (52) Empty reply from server
[17:24:39] <klausman>	 (from deploy2002)
[17:25:26] <klausman>	 (ah, https, nvm)
[17:40:02] <klausman>	 elukey: The IP restbase is on is already in our list of allowed IPs, right?
[17:40:28] <elukey>	 klausman: ah no I thought network policies were added!
[17:40:40] <elukey>	 that's it then, sorry I assumed the wrong thing
[17:40:49] <klausman>	 I can't remember where the IP ranges are configured
[17:41:05] <elukey>	 it is in the article description's value.yaml under helmfile
[17:41:21] <elukey>	 you'll find also api-ro etc..
[17:41:35] <elukey>	 you can also edit networkpolicies in the pods namespace
[17:42:00] <elukey>	 to quickly check
[17:42:16] <elukey>	 or do you mean the ips for the service itself? (discovery one)
[17:42:40] <elukey>	 if so netbox, but a DNS lookup would do it as well (we don't change the VIP IPs once created, so we are good)
[17:43:37] <klausman>	 helmfile.d/ml-services/experimental/values.yaml seems like the right place
[17:45:05] <elukey>	 exactly yes
[17:45:47] <klausman>	 mh. looks like there isn't one in codfw, only eqiad
[17:46:10] <klausman>	 nvm, typo
[17:46:17] <elukey>	 ahhh okok I was worried :D
[17:50:05] <klausman>	 It works!
[17:50:13] <klausman>	 >>> r.content
[17:50:15] <klausman>	 b'{"type":"standard","title":"Clandonald","displayti
[17:50:17] <klausman>	 (truncated)
[17:50:25] <klausman>	 Review has been updated
[17:51:09] <elukey>	 nice!
[17:52:39] <elukey>	 +1
[17:52:59] <klausman>	 merci :)
[17:53:11] <klausman>	 I will merge and check it is a noop over my already-done edits.
[17:58:42] <klausman>	 all good
[17:58:58] <elukey>	 klausman: do you want to explain what we did to the team tomorrow?
[17:59:00] <klausman>	 kevinbazira: the service should now work, should be good for your tests tomorrow
[17:59:46] <klausman>	 yep
[18:03:14] <kevinbazira>	 thank you for fixing the Istio mesh klausman and elukey. is this endpoint working now:
[18:03:14] <kevinbazira>	 ```
[18:03:14] <kevinbazira>	 http://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/page/summary/Clandonald
[18:03:14] <kevinbazira>	 ```
[18:03:15] <kevinbazira>	 if yes, in the model-server we were using:
[18:03:16] <kevinbazira>	 ```
[18:03:16] <kevinbazira>	 http://api-ro.discovery.wmnet/api/rest_v1/page/summary/Clandonald
[18:03:17] <kevinbazira>	 ```
[18:03:17] <kevinbazira>	 we shall push a patch that updates it to:
[18:03:18] <kevinbazira>	 ```
[18:03:18] <kevinbazira>	 http://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/page/summary/Clandonald
[18:03:19] <kevinbazira>	 ```
[18:09:27] <elukey>	 kevinbazira: the port is 4111 but the rest is good 
[18:09:54] <elukey>	 so one thing to keep in mind - in this case, en.wikipedia.org acts as host header
[18:10:01] <elukey>	 I don't know if we need to support more
[18:10:19] <elukey>	 so the parameter may need to vary and not be hardcoded
[18:11:12] * elukey afk!
[18:11:19] <elukey>	 have a nice rest of the day folks!
[18:12:07] <kevinbazira>	 elukey: thank you for clarifying on the port. we shall use `4111`
[18:13:16] <kevinbazira>	 regarding the host headers, the model-server supports about 24 languages no only `en`:
[18:13:16] <kevinbazira>	 https://github.com/wikimedia/machinelearning-liftwing-inference-services/blob/main/article-descriptions/model-server/utils.py#L11-L37 
[18:14:35] <kevinbazira>	 we shall push the patch tomorrow. enjoy the rest of your day. o/
[19:14:54] <wikibugs>	 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Growth-Team, 10Wikipedia-Android-App-Backlog, 10Patch-For-Review: Add revertrisk-language-agnostic to RecentChanges filters - https://phabricator.wikimedia.org/T348298 (10KStoller-WMF) > In T348298#9362986, @Samwalton9-WMF wrote: > This patch also a...
[20:13:35] <wikibugs>	 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Growth-Team, 10Wikipedia-Android-App-Backlog: CRS: Community rollout plan and discussion about adding revertrisk to RecentChanges filters - https://phabricator.wikimedia.org/T352217 (10KStoller-WMF)
[20:14:00] <wikibugs>	 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Growth-Team, 10Wikipedia-Android-App-Backlog: CRS: Community rollout plan and discussion about adding revertrisk to RecentChanges filters - https://phabricator.wikimedia.org/T352217 (10KStoller-WMF) p:05Triage→03Medium
[23:00:52] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Sprint 3 (Growth Team)), 10User-notice: Deploy "add a link" to 15th round of wikis - https://phabricator.wikimedia.org/T308141 (10Etonkovidova)
[23:04:05] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Sprint 3 (Growth Team)), 10User-notice: Deploy "add a link" to 15th round of wikis - https://phabricator.wikimedia.org/T308141 (10Etonkovidova) Checked some wikis from the list - all looks as expected. Leaving the task in the Test in Production columns to...
[23:05:25] <wikibugs>	 10Machine-Learning-Team, 10Product-Analytics, 10User-Iflorez: Transient error while running lift wing topic model - https://phabricator.wikimedia.org/T351114 (10Iflorez) 05Open→03Resolved
[23:13:34] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Sprint 3 (Growth Team)), 10Serbian-Sites, and 3 others: Deploy "add a link" to 16th round of wikis - https://phabricator.wikimedia.org/T308142 (10Etonkovidova)
[23:15:21] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Sprint 3 (Growth Team)), 10Serbian-Sites, and 3 others: Deploy "add a link" to 16th round of wikis - https://phabricator.wikimedia.org/T308142 (10Etonkovidova) Checked selected wikis from the list - leaving in the Test in Production column for monitoring...
[23:19:52] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Sprint 3 (Growth Team)), 10Turkish-Sites, 10User-notice: Deploy "add a link" to 17th round of wikis - https://phabricator.wikimedia.org/T308143 (10Etonkovidova)
[23:21:27] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Sprint 3 (Growth Team)), 10Turkish-Sites, 10User-notice: Deploy "add a link" to 17th round of wikis - https://phabricator.wikimedia.org/T308143 (10Etonkovidova) Checked selected wikis from the list - all works as expected; leaving in the **Test in Produ...