[06:44:48] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10elukey) After a chat with @daniel on slack we realized that the endpoint is indeed used: https://w.wiki/6rKU @akosiaris I don't think... [10:03:43] I'd like to ease pressure on thumbor a bit by reducing the volume of thumbnail generation requests from changeprop https://gerrit.wikimedia.org/r/931568 [10:28:09] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10akosiaris) >>! In T338471#8947818, @elukey wrote: > After a chat with @daniel on slack we realized that the endpoint is indeed used: ht... [11:01:00] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10daniel) Wuld it be possible to implement a compatibility layer, so the app can use the new service without any changes needed? Updating... [12:25:13] 10serviceops, 10Wikimedia Enterprise, 10Performance-Team (Radar), 10affects-Kiwix-and-openZIM: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866 (10Ladsgroup) One of the reasons that it hits production to the size of 100s millions of times a month i... [12:33:35] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10elukey) @diego we probably need to figure out a path forward for this, namely: 1) review how the old recommendation-api works, what tr... [12:54:32] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10diego) >>! In T338471#8948653, @elukey wrote: > @diego we probably need to figure out a path forward for this, namely: > > 1) re... [13:15:34] 10serviceops, 10Thumbor, 10Patch-For-Review: Thumbor's use of poolcounter is rate limiting Kubernetes IPs - https://phabricator.wikimedia.org/T339863 (10Krinkle) > However, lots of requests from changeprop seem likely in light of T337649. Do you mean jobrunners (mw appservers, PHP) or changeprop itself? If... [13:15:56] 10serviceops, 10Thumbor, 10Patch-For-Review, 10Performance-Team (Radar): Thumbor's use of poolcounter is rate limiting Kubernetes IPs - https://phabricator.wikimedia.org/T339863 (10Krinkle) [14:12:23] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10elukey) @diego in https://phabricator.wikimedia.org/T308165#7983559 @Isaac also mentioned this: > For example, similar service with en... [14:32:15] 10serviceops, 10Thumbor, 10Patch-For-Review, 10Performance-Team (Radar): Thumbor's use of poolcounter is rate limiting Kubernetes IPs - https://phabricator.wikimedia.org/T339863 (10hnowlan) >>! In T339863#8948816, @Krinkle wrote: >> However, lots of requests from changeprop seem likely in light of T337649.... [17:48:43] jayme: elukey btullis, i was about to apply https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/927224 to delete the stream-enrichment-poc namespace from dse-k8s-eqiad, but it looks like there is an unapplied bump to the calico chart version [17:49:07] trying to limit to release=namespaces doesn't seem to work [18:18:31] ottomata: From my perspective, I'd say go ahead and deploy. There's still nothing I know of running on the cluster, other than the spark-operator which isn't currently being used. elukey did some tests of GPU access, but I don't think they're currently running. [18:25:18] okay [18:30:58] done [18:39:35] 👍