[06:47:05] <wikibugs>	 (03CR) 10Kevin Bazira: [C: 03+1] revscoring: fix exception handling in fetch_features [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939321 (owner: 10Elukey)
[06:56:59] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C: 03+1] revscoring: fix exception handling in fetch_features [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939321 (owner: 10Elukey)
[06:57:51] <elukey>	 hello folks :)
[06:57:57] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] revscoring: fix exception handling in fetch_features [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939321 (owner: 10Elukey)
[06:58:32] <isaranto>	 o/
[07:04:22] <wikibugs>	 (03Merged) 10jenkins-bot: revscoring: fix exception handling in fetch_features [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939321 (owner: 10Elukey)
[07:11:22] <wikibugs>	 10Machine-Learning-Team, 10MediaWiki-extensions-ORES: Deployment of Lift Wing usage to all wikis that use ores extension - https://phabricator.wikimedia.org/T342115 (10matej_suchanek)
[07:53:16] <elukey>	 isaranto: o/
[07:53:54] <isaranto>	 o/
[07:54:06] <elukey>	 I was revieweing the aiohttp's client session docs and IIUC, by default, when a new connection is opened the Keep alive http header is added
[07:54:37] <elukey>	 in our case, we tore down the connection every time via the context manager, but I am wondering if this is causing envoy to keep some broken connections around
[07:55:01] <elukey>	 it is a stretch but I'd test it to see if anything improves
[07:55:51] <elukey>	 basically adding connector=aiohttp.TCPConnector(force_close=True)
[07:56:21] <elukey>	 I'd also set use_dns_cache=False
[07:56:45] <klausman>	 Morning!
[07:56:53] <elukey>	 morning :)
[07:57:01] <klausman>	 elukey: alert is firing for ml-s-1007's mgmt card :-/ Investigating
[07:57:13] <elukey>	 ack
[08:02:33] <isaranto>	 elukey: ack
[08:02:54] <klausman>	 Ah, DCOps already has a ticket with a whole load of hosts, all in rack F2. I suspect a switch there fell over.
[08:03:05] <klausman>	 I'll subscribe to that ticket and keep track of it
[08:03:37] <wikibugs>	 (03PS1) 10Elukey: ores-legacy: reduce connection handling for aiohttp [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939626 (https://phabricator.wikimedia.org/T341479)
[08:03:48] <elukey>	 klausman: nice!
[08:03:53] <klausman>	 ms1005 has high latency, might that be related to the stuck-ness of pods yesterday?
[08:03:53] <elukey>	 isaranto: --^
[08:04:20] <elukey>	 since when? Some minutes or more?
[08:04:34] <klausman>	 5-10m ago
[08:04:41] <isaranto>	 on it
[08:04:49] <klausman>	 https://alerts.wikimedia.org/?q=alertname%3DKubeletOperationalLatency&q=team%3Dsre&q=%40receiver%3Ddefault
[08:04:52] <isaranto>	 sry I mean Luca's patch
[08:04:58] <elukey>	 sometimes it is transient, shouldn't be related to yesterday 
[08:05:12] <elukey>	 but you can check what's wrong, maybe it is a specific thing that doesn't work
[08:05:15] <elukey>	 :)
[08:05:42] <elukey>	 (my suspicion is that the alerts are a little sensitive)
[08:05:46] <klausman>	 I will do some looking around, that will surely make it go away :)
[08:07:27] * elukey bbiab
[08:15:50] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C: 03+1] ores-legacy: reduce connection handling for aiohttp [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939626 (https://phabricator.wikimedia.org/T341479) (owner: 10Elukey)
[08:19:19] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C: 03+1] ores-legacy: reduce connection handling for aiohttp (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939626 (https://phabricator.wikimedia.org/T341479) (owner: 10Elukey)
[08:22:39] <isaranto>	 I am attempting a refactoring of inf services repo so that we can run model servers without packaging in docker so that we can try stuff out fast
[08:23:06] <isaranto>	 I'll abandon if it requires more than 1-2 h of work
[08:28:29] <wikibugs>	 (03CR) 10Elukey: ores-legacy: reduce connection handling for aiohttp (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939626 (https://phabricator.wikimedia.org/T341479) (owner: 10Elukey)
[08:28:31] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] ores-legacy: reduce connection handling for aiohttp [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939626 (https://phabricator.wikimedia.org/T341479) (owner: 10Elukey)
[08:29:23] <wikibugs>	 (03Merged) 10jenkins-bot: ores-legacy: reduce connection handling for aiohttp [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939626 (https://phabricator.wikimedia.org/T341479) (owner: 10Elukey)
[08:30:03] <isaranto>	 I hope u don't hate me after this :)
[08:33:16] <elukey>	 never <3
[08:37:06] <klausman>	 elukey: so I have not found yet what makes 1005 slow, but it's not the only kubelet that has rather suddenly started slwoing down starting yesterday noon UTC
[08:37:16] <klausman>	 (possibly earlier, it's hard to see
[08:37:18] <klausman>	 )
[08:37:27] <klausman>	 https://grafana.wikimedia.org/goto/CDDRpwjVz?orgId=1
[08:47:20] <elukey>	 isaranto: it seems to have improved things a lot!
[08:47:40] <isaranto>	 aha! that's nice!
[08:47:53] <isaranto>	 is it deployed? can I check?
[08:48:12] <elukey>	 isaranto: in staging yes
[08:48:33] <isaranto>	 did u see the same issues in staging before?
[08:48:41] <elukey>	 I did yes
[08:48:46] <isaranto>	 ok!
[08:48:52] <elukey>	 I tested it only with goodfaith and a lot of rev ids
[08:49:04] <elukey>	 my test call now is better
[08:49:13] <elukey>	 not sure though the huge one in the task, haven't tried it yet
[08:49:45] <elukey>	 klausman: nice finding! Does it match with us deploying? Because we rolled out the new concurrency stuff yesterday IIRC for knative
[08:50:50] <elukey>	 yeah 1005 seems the most affected one
[08:51:02] <elukey>	 lemme check the eqiad pods
[08:53:17] <elukey>	 klausman: ah wow on 1005 dmesg looks really messy
[08:53:19] <elukey>	 sigh
[08:56:55] <elukey>	 isaranto: super big queries still lead to some 503s
[08:57:18] <elukey>	 but it seems that we are doing better afaics
[08:57:31] <klausman>	 elukey: yeah, lots of OOM kills.
[08:57:58] <klausman>	 Though the machine is not under memory pressure right now.
[08:58:11] <klausman>	 I wonder if the OOM kills got it into a wonky state
[08:58:22] <elukey>	 yeah I'd restart the kubelet to be honest
[08:58:33] <klausman>	 already done that, also kubeproxy, no effect
[08:59:25] <elukey>	 let's be careful in restarting those daemons, kubelet should be safe to restart, not sure if kubeproxy needs a drain first etc..
[08:59:29] <klausman>	 Also note the ethernet flaps
[09:00:23] <klausman>	 kubeproxy is fine to restart as well, AIUI
[09:01:09] <klausman>	 Also, it seems the other kubelets are starting to go up in latency as well right now
[09:01:20] <elukey>	 there are also SATA link down events
[09:02:54] <klausman>	 those are boot messages
[09:02:57] <elukey>	 nothing reported in racadm's getsel
[09:03:28] <klausman>	 The machine booted on May 19, 13:53, and the SATA link messages are from that time (dmesg -T)
[09:03:40] <elukey>	 ahhh right my pebcak, I've read 19 and not the month
[09:04:16] <klausman>	 NETDEV_CHANGE _may_ be caused by external factors, but I've never seen a link flap
[09:04:46] <klausman>	 oh dear, the API latency is skyrocketing. ?1s now
[09:04:51] <klausman>	 >1s
[09:08:03] <elukey>	 ?
[09:08:14] <klausman>	 https://grafana.wikimedia.org/goto/TdBU2Qj4k?orgId=1
[09:08:21] <klausman>	 It has since recovered.
[09:08:28] <klausman>	 But likely not completely
[09:08:38] <elukey>	 that's not the API though :)
[09:08:56] <klausman>	 ok, op latency
[09:09:25] <elukey>	 ok so let's start from what we know
[09:09:43] <elukey>	 some kubelets started to show hugh latencies since yesterday around 9:20 UTC
[09:09:46] <elukey>	 what changed?
[09:09:54] <klausman>	 Starting yesterday around/just before noon, Kubelt op latency is much higher than before.
[09:09:55] <isaranto>	 I'm seeing some issues on revscoring-editquality-goodfaith namespace on eqiad. There are some enwiki pods that fail witha a backoff. It seems the same issue as yesterday they are using an older revision
[09:10:51] <elukey>	 ok let's also check the events there, maybe k8s is giving us some indication
[09:10:51] <klausman>	 There are some pods that have ~200 restarts, and one is in crashloop backoff (6200 restarts)
[09:11:01] <elukey>	 this is more promising, what pods?
[09:11:13] <klausman>	 nllb-200-predictor-default-00005-deployment-6c68c75bcf-chtnc is the crashloop backoff one
[09:11:34] <klausman>	 The rest are all enwiki-goodfaith-predictor-default
[09:11:46] <klausman>	 one enwiki-drafttopic-predictor-default is in terminating state
[09:12:09] <klausman>	 All pods are on diverse machines, except the nllb and one of the goodfaith that are both on 1001
[09:12:39] <elukey>	 the most worrying ones are the istio gateway restarts in my opinion
[09:13:15] <klausman>	 Yes, those are slower (freq, but more impactful
[09:13:16] <elukey>	 klausman: let's start cleaning up the goodfaith ones as isaranto suggested
[09:14:34] <elukey>	 isaranto started the deployments at around 09:16 UTC yesterday, so the new code is definitely related https://sal.toolforge.org/log/SJ1JaIkBxE1_1c7scKjt
[09:15:21] <elukey>	 and most of the high latencies (as seen in https://grafana.wikimedia.org/d/000000472/kubernetes-kubelets?var-cluster=eqiad%20prometheus%2Fk8s-mlserve&orgId=1&from=now-2d&to=now&refresh=30s&viewPanel=25) seem to be related to "stop container"
[09:15:59] <isaranto>	 TIL about sal.toolforge.org
[09:16:30] <klausman>	 I've taken a quick look at the istio pod logs, but nothing immediately obvious is wrong there
[09:17:29] <klausman>	 What's the cleanup procedure for the goodfaith pods?
[09:18:13] <elukey>	 klausman: this is the corner case of old knative revisions getting stuck, it may be a bug in our version. What I usually do is "kubectl get revision -n $namespace" and then delete the correspondent one
[09:18:24] <elukey>	 you can find the messy revision checking the name of the pod
[09:19:17] <klausman>	 I see 8 9 and 10
[09:19:40] <klausman>	 But only 9 and 10 have desired replicas=4
[09:19:47] <isaranto>	 9 is the messy one in this case
[09:20:05] <elukey>	 on ml-serve1005
[09:20:05] <elukey>	 Jul 19 09:16:14 ml-serve1005 kubelet[3136643]: E0719 09:16:14.417212 3136643 remote_runtime.go:479] "StopContainer from runtime service failed" err="rpc error: code = Unknown desc = operation timeout: context deadline exceeded" containerID="6b89b3c157d64bbec91e96e7d7b9c6c51ac47a2120719906fda0bf487b308396"
[09:20:09] <isaranto>	 that is probably the issue. 9 shouldn't have desired replicas
[09:20:48] <klausman>	 So `kubectl delete revision -n revscoring-editquality-goodfaith enwiki-goodfaith-predictor-default-00009` ?
[09:21:01] <elukey>	 yep
[09:21:14] <klausman>	 Ok, it's gone
[09:21:37] <klausman>	 It's now in state terminating
[09:22:14] <klausman>	 One pod left
[09:24:53] <klausman>	 The question is: how did we get into this state, and why does it wreak havoc on kubelet op latency
[09:25:27] <klausman>	 (the latter is probably due to the DELET being slow, and frequent crashlooping making it the dominating operation, staistically)
[09:26:12] <elukey>	 we know from the graph above that the op latency issues are related to the stop container 
[09:26:26] <klausman>	 (last pod gone)
[09:26:53] <klausman>	 The op latency graph _looks_ recovered, but it will take another 15m-30m to be sure.
[09:26:58] <elukey>	 isaranto: nllb status looks weird, seems failing to bootstrap with the GPU, any experiment ongoing over there?
[09:28:00] <elukey>	 need to run for an errand, bbl!
[09:30:08] <isaranto>	 I'm taking a look, dont really remember the state I left it
[09:33:21] <elukey>	 (false alert, back :)
[09:34:36] <klausman>	 There are two revisisons for nllb: rev 2 and rev 5
[09:34:38] <isaranto>	 yes, I had left nllb hanging while experimenting with quantization
[09:34:39] <isaranto>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/939637
[09:34:43] <klausman>	 rev2 is running, rev5 is in Error
[09:34:57] <isaranto>	 I set quantization to false and also removed the gpu for now
[09:35:43] <isaranto>	 sry folks totally forgot about that one
[09:36:21] <elukey>	 np, it is surely not the problem
[09:36:53] <elukey>	 it started when you did all deployments yesterday, so I guess that it is either the knative change or maybe the broad deployments that set some weird bug off
[09:41:53] <elukey>	 klausman: as you mentioned latencies seem to have recovered a bit, I still don't explain the OOMs that we found though, maybe unrelated but worrying
[09:41:58] <elukey>	 and also all the gateway restarts
[09:44:10] <isaranto>	 I deployed the new changes in nllb
[09:44:52] <wikibugs>	 (03PS1) 10Elukey: ores-legacy: set connection limit to 0 (unlimited) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939639 (https://phabricator.wikimedia.org/T341479)
[09:45:13] <isaranto>	 it is running fine now but that also has some old revision, along with bloom-560
[09:45:25] <klausman>	 elukey: yeah, Latency looks good. As for the ooms, I'lls if there's a pattern to the processes killed, and if it happened on any of the other machines
[09:46:18] <wikibugs>	 (03PS2) 10Elukey: ores-legacy: set connection limit to unlimited [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939639 (https://phabricator.wikimedia.org/T341479)
[09:46:47] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: [C: 03+1] ores-legacy: set connection limit to unlimited [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939639 (https://phabricator.wikimedia.org/T341479) (owner: 10Elukey)
[09:47:08] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] ores-legacy: set connection limit to unlimited [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939639 (https://phabricator.wikimedia.org/T341479) (owner: 10Elukey)
[09:47:38] <elukey>	 uff I ran tix
[09:47:40] <elukey>	 *tox
[09:47:58] <wikibugs>	 (03PS3) 10Elukey: ores-legacy: set connection limit to unlimited [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939639 (https://phabricator.wikimedia.org/T341479)
[09:48:28] <klausman>	 All machines have some OOMs in their dmesg, typically 2-3 a day, and the majority of them, the process name is envoy
[09:49:10] <elukey>	 thanks for the review isaranto <3
[09:49:21] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] ores-legacy: set connection limit to unlimited [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939639 (https://phabricator.wikimedia.org/T341479) (owner: 10Elukey)
[09:49:33] <klausman>	 A typical dmegs entry is `[Wed Jul 19 08:58:53 2023] Memory cgroup out of memory: Killed process 4057687 (envoy)`
[09:49:51] <klausman>	 plsu some memstats. Unfortunately, that doesn't tell us what specific envoy/pod it was
[09:50:12] <wikibugs>	 (03Merged) 10jenkins-bot: ores-legacy: set connection limit to unlimited [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939639 (https://phabricator.wikimedia.org/T341479) (owner: 10Elukey)
[09:50:19] <elukey>	 klausman: https://grafana.wikimedia.org/d/hyl18XgMk/kubernetes-container-details?orgId=1&var-datasource=eqiad%20prometheus%2Fk8s-mlserve&var-namespace=istio-system&var-pod=istio-ingressgateway-c7z5j&var-container=All 
[09:50:23] <elukey>	 sigh
[09:50:32] <elukey>	 it takes nothing to get ooms in this stage
[09:50:37] <elukey>	 *state
[09:50:50] <elukey>	 the limit is too low
[09:50:52] <klausman>	 Question is: are we under-provisioning or is there a memory problem in istio
[09:51:36] <klausman>	 Looking at the history, istio goes from start to 90+% of mem usage very quickly. But modern Go programs tend to do that.
[09:51:57] <klausman>	 (assuming they are setup that way)
[09:52:20] <isaranto>	 elukey: do u think 503 errors will go await by setting limit to inf? I doubt that we reached the limit of 100
[09:52:43] <isaranto>	 regardless though it is a good change - to not control things on lw side
[09:52:43] <klausman>	 elukey: should we bump the mmemory allowance for the ingress GWs?
[09:52:52] <elukey>	 klausman: +1 yes
[09:52:59] <klausman>	 I'll make a patch
[09:53:43] <elukey>	 isaranto: so the only 503s that I get is when I issue the big request that you added in the  task, and judging from the code we loop through models and rev_ids within the aiohttp's Client session
[09:54:18] <elukey>	 rev-ids * 4 models is more than 100
[09:54:33] <elukey>	 does it make sense?
[09:55:32] <isaranto>	 yes, but they go to different pods/model-ervers
[09:56:33] <elukey>	 sure sure, but limit's doc says "total number simultaneous connections", IIUC this is a setting for each ClientSession
[09:56:47] <elukey>	 so it is agnostic to the targets
[09:56:59] <elukey>	 what it cares about is how many tcp connections we open
[09:57:06] <isaranto>	 nevermind I got confused
[09:57:20] <elukey>	 nono please let's discuss, I am not 100% sure yet
[09:57:23] <isaranto>	 I was thinking server side (lw)
[09:57:33] <elukey>	 ahh okok
[09:57:41] <elukey>	 I think we should apply these settings to all model servers
[09:57:51] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/939640/
[09:58:04] <isaranto>	 I mean it makes sense all connections are from ores-legacy. I see 49 revids and 4 models ~=200 requests
[09:58:13] <elukey>	 yeah exactly
[09:58:38] <elukey>	 and a brutal 503 seems to fit the picture if you reach that number
[09:58:58] <isaranto>	 but! shouldnt we have the same issue when I run this on statbox?
[09:59:48] <elukey>	 in theory no since you don't have the envoy local proxy
[09:59:59] <elukey>	 you connect directly to lift wing, so no interference
[10:00:05] <elukey>	 and aiohttp is free to handle is conn
[10:00:14] <elukey>	 (the 503s come from the ores-legacy's tls proxy_
[10:01:19] <isaranto>	 ok ok, thnx for clarifying
[10:02:06] <klausman>	 elukey: I am having trouble finding the right file/spot for the quota increase. helmfile.d/admin_ng/values/ml-serve.yaml seemed right, but I'm not sure anymore
[10:02:34] <elukey>	 klausman: ah yes with istio is a little bit more complex, since we use a separate manifeste
[10:02:40] <elukey>	 *manifest
[10:02:56] <elukey>	 it is in custom.d, and I think there should be an option to tune the memory for the gateway pods
[10:03:01] <elukey>	 if not we'd need to check in their docs
[10:03:55] <klausman>	 custom_deploy.d?
[10:04:19] <elukey>	 yes exactly
[10:04:44] <elukey>	 I am going to lunch with Filippo today, so I am going afk earlier for lunch, ttl!
[10:04:49] * elukey lunch
[10:05:00] <klausman>	 \o do say hello to Filippo :)
[10:50:23] <klausman>	 Aaand latency on 1005 (but only there) is up again. Investigating.
[10:53:40] <klausman>	 Something weird is going on with enwiki-drafttopic-predictor
[10:54:13] <klausman>	 desired replicas is 1, but there is one running (25h old) and another terminating. And a few mins ago there was yet another in init.
[10:54:27] <klausman>	 The currently terminating one is 2m45s odl. 
[10:54:34] <klausman>	 And they're all rev 9
[10:54:50] <klausman>	 (I am focusing on that pod since it's running on 1005)
[10:57:37] <klausman>	 aand now there are three again. Three pods, that is not 3/3 containers.
[11:01:46] <klausman>	 elukey: See above. Something weird is going on there. Ther eis a 25h old pod in steady state, and then two more that are cycling through init/termination all the time. They're all rev 9, so it isn't the same stuff me had earlier.
[11:03:14] <klausman>	 oooh, there is a desired replicas=3
[11:03:33] <klausman>	 And how we have three running and a fourth terminating.
[11:03:41] <klausman>	 Is this maybe autoscaling being twitchy?
[11:04:09] <klausman>	 Yes, desired replica is going up and down a lot.
[11:11:38] <isaranto>	 I see they are all from the latest revision. desired replica shouldn't be 3 I guess
[11:11:47] <isaranto>	 https://www.irccloud.com/pastebin/tP9iv0P2/
[11:11:57] <klausman>	 I think it might be autoscaling
[11:12:24] <isaranto>	 what happens if you edit the inference service for drafttopic and set maxReplicas to 1? can u check if that would resolve it?
[11:12:26] <klausman>	 if it is, maybe it should keep started pods around for longer
[11:13:11] <klausman>	 The other question is if this up/down of replicas is actually a problem (and if the op latency alert is too sensitive)
[11:13:30] <klausman>	 I'll wait for Luca to come back before I fiddle with things. Also: lunch :)
[11:14:13] <isaranto>	 ack
[11:14:23] <isaranto>	 I think the issue may come from `autoscaling.knative.dev/target: "3"`
[11:19:04] <isaranto>	 although again this refers to the number of concurrent requests that should trigger a scaleup
[11:19:58] <isaranto>	 it is quite confusing and I forgot even what we were discussing yesterday :P
[11:22:17] <isaranto>	 concurrent requests vs replicas. So knative will create a new pod if we have more than 3 concurrent requests in 1 pod and that will happen max until we reach 3 pods. all set by these annotations
[11:22:17] <isaranto>	 ```
[11:22:17] <isaranto>	      autoscaling.knative.dev/max-scale: 3
[11:22:17] <isaranto>	      autoscaling.knative.dev/min-scale: 1
[11:22:17] <isaranto>	      autoscaling.knative.dev/target: 3
[11:22:18] <isaranto>	 ```
[11:22:39] <isaranto>	 I'm just writing down my conclusions to make sure I'm getting it right
[11:22:48] * isaranto goes for lunch
[11:25:37] <klausman>	 It sounds about right
[11:57:10] <elukey>	 checking
[12:00:02] <elukey>	 klausman: one useful source of info is "kubectl get events -n etc.."
[12:00:49] <klausman>	 ah, good point
[12:01:02] <elukey>	 some issues seems to be caused by the readiness probe getting a 503
[12:02:22] <klausman>	 I made a patch for the istio memory, feel free to review and merge, I gtg to a doc appt. Should be back before the meetings
[12:02:55] <elukey>	 ack
[12:25:03] <elukey>	 applying your changes :)
[12:27:02] <elukey>	 all right all good, I see the changes applied etc..
[12:27:16] <elukey>	 so we'll keep the restarts monitored
[12:56:07] <elukey>	 isaranto: I think I was doing wrong tests before, the situation is a little better now but the big query in staging fails with a lot of timeouts
[12:56:10] <elukey>	 sigh
[12:56:29] * isaranto sighs as well
[12:58:02] <elukey>	 I forgot about https://logstash.wikimedia.org/app/dashboards#/view/138271f0-40ce-11ed-bb3e-0bc9ce387d88?_g=h@c823129&_a=h@780154f, but we have access logs breakdown at the istio gateway level
[12:59:58] <elukey>	 if I isolate the Ores legacy UA, I don't see many requests to the istio gateway pods in ml-serve-codfw (we use the tls local proxy that calls production, even from staging)
[13:00:09] <elukey>	 and all of them are 200
[13:00:15] <Amir1>	 isaranto: hey, which ores stuff is blocked on me?
[13:00:33] <Amir1>	 (i.e. what can I do to help?)
[13:01:34] <isaranto>	 Amir1: I'm planning on rolling this out to all wikis leaving enwiki last. 
[13:01:59] <isaranto>	 I can create the patch(es) and you could deploy since I can't do that part
[13:03:56] <isaranto>	 and we need a checklist for things to check after each deployment to make sure things work. e.g. run a job on mwmaint, check database entries , recentchanges filters etc
[13:03:57] <isaranto>	 wdyt?
[13:10:21] <Amir1>	 sure, let's start with one small wiki and move forward
[13:14:20] <wikibugs>	 (03PS1) 10Ilias Sarantopoulos: refactor: allow relative imports in repo [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939683
[13:14:27] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] refactor: allow relative imports in repo [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/939683 (owner: 10Ilias Sarantopoulos)
[13:15:54] <isaranto>	 the above is WIP, plz disregard, I have an issue with appending to PYTHONPATH with blubber and reached out to releng
[13:18:14] <isaranto>	 Amir1: any good candidates from the table here -> https://phabricator.wikimedia.org/T342115?
[13:19:32] <Amir1>	 first eswikibooks and eswikiquote. Then hewiki and itwiki (they are goup1, as an early adapter), ...
[13:28:14] <elukey>	 isaranto: trying to rollback the limit change, and see if it improves
[13:28:18] <elukey>	 (only in deployment-charts)
[13:28:54] <isaranto>	 ok!
[13:57:43] <isaranto>	 https://www.irccloud.com/pastebin/UTL7uNMM/
[14:38:59] <klausman>	 elukey: for reference, get events cmdline that works better than defaults: kubectl get events -w -o custom-columns=Time:.firstTimestamp,Component:.source.component,Type:.type,Reason:.reason,Message:.message -n revscoring-drafttopic 
[14:45:52] <elukey>	 okok thanks
[14:54:41] <elukey>	 created https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/939719 and next to add the scale down delay to knative
[14:54:44] <elukey>	 :)
[14:54:54] <elukey>	 (already applied it manually in eqiad)
[14:55:11] <elukey>	 latencies are still weird, but one poid is in terminating state
[15:07:50] <klausman>	 LGTM for 939719
[15:18:15] <elukey>	 lol I just realized from the ci's diff that we have some logging config
[15:18:24] <elukey>	 never really applied since "config_logging" wasn't there
[15:18:39] <elukey>	 basically
[15:18:40] <elukey>	 /home/elukey/Wikimedia/deployment-charts/charts/knative-serving/values.yaml
[15:18:43] <elukey>	 sorry
[15:18:46] <elukey>	   config_logging:
[15:18:46] <elukey>	     loglevel.controller: "warn"
[15:18:47] <elukey>	     loglevel.autoscaler: "warn"
[15:18:47] <elukey>	 etc..
[15:19:04] <elukey>	 it is in the knative's chart values.yaml already
[15:19:31] <elukey>	 it should be ok to apply, so we remove some weird logs 
[15:23:08] <wikibugs>	 10Machine-Learning-Team: use wikiID in inference name on LW for revscoring models - https://phabricator.wikimedia.org/T342266 (10isarantopoulos)
[15:27:37] <klausman>	 elukey: nice catch
[15:28:57] <elukey>	 klausman: nice pebcak of Luca from the past I'd say :D
[15:29:18] <elukey>	 pretty sure I added it
[15:29:21] <elukey>	 sigh
[15:29:26] <klausman>	 it's all glass-half-empty vs full etc :)
[15:29:55] <elukey>	 sure sure :)
[15:30:02] <elukey>	 I am deploying the knative changes to all clusters
[15:31:47] <klausman>	 :+1:
[15:32:28] <elukey>	 aaand it fails
[15:33:07] <elukey>	 afaics changing the log settings to all pods knocks the webhook down, so the new yalm is not validated
[15:33:11] <elukey>	 and it is rejected
[15:33:22] * elukey cries in corner
[15:33:52] <klausman>	 That seems like a serious design byg
[15:38:44] <elukey>	 filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/939733
[15:38:48] <elukey>	 as workaround
[15:38:51] <elukey>	 (for the moment)
[15:39:31] <klausman>	 lgtm'd
[15:49:24] <elukey>	 danke
[16:03:49] <wikibugs>	 10Machine-Learning-Team: Append PYTHONPATH in blubber - https://phabricator.wikimedia.org/T342273 (10isarantopoulos)
[16:04:16] <isaranto>	 o/ I added a task for a blubber improvement relevant to the refactoring I tried today
[16:11:10] <elukey>	 nice!
[16:15:49] <isaranto>	 I've already opened the MR, but I'll just test it first https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/49
[16:30:07] <elukey>	 klausman: ahhh ok the webhook thing is due to
[16:30:07] <elukey>	 Readiness probe failed: HTTP probe failed with statuscode: 503
[16:30:15] <elukey>	 so the new pods are probably slow to come up
[16:31:32] <elukey>	     Liveness:   http-get https://:8443/ delay=120s timeout=1s period=1s #success=1 #failure=6
[16:31:35] <elukey>	     Readiness:  http-get https://:8443/ delay=0s timeout=1s period=1s #success=1 #failure=3
[16:31:38] <elukey>	 yeah this needs to change
[16:31:50] <klausman>	 Ah, so we need to increas delay (and possibly timeout) for readiness?
[16:31:57] <elukey>	 exactly yes
[16:34:07] <isaranto>	 I'm battling with helm in this patch https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/939744
[16:34:40] <isaranto>	 I'm logging off and will look at it with fresh eyes and mind tomorrow
[16:34:57] <isaranto>	 logging off for the day
[16:35:45] <elukey>	 me too :)
[16:35:52] <klausman>	 \o
[16:38:08] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/939745 should fix the readiness probe
[16:39:13] <klausman>	 That verified+1 is a misclick, but LGTM!
[16:48:03] <elukey>	 so weird, still doesn't work
[16:48:13] <elukey>	 I'll try to work on it tomorrow :)
[16:48:14] <elukey>	 o/