[07:27:28] morning! [07:27:41] I may have found the issue with knative, firewall - https://gerrit.wikimedia.org/r/c/operations/puppet/+/724933 [07:27:49] once reviewed merged it should be unblocked [10:28:33] Error creating: pods "enwiki-goodfaith-predictor-default-jnb8s-deployment-b86764pvxb9" is forbidden: minimum cpu usage per Container is 100m, but request is 25m [10:28:37] this is the new error :D [10:30:15] that is for the sidecar container, and I think by default we have limits in the namespace [10:30:30] it is weird since in this case we are asking less than the min [11:16:06] ok the pods are coming up! But failing.. [11:16:25] it seems a problem with the service account that accesses the s3 secret, will have to work on it [11:31:32] * elukey lunch [12:52:25] 10Machine-Learning-Team, 10Observability-Logging: logstash indexing failures reported for "knative-serving" - https://phabricator.wikimedia.org/T292174 (10fgiunchedi) [14:50:05] so https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/725001 may be the solution [14:50:15] [popcorn emoji] [14:50:17] but I'd need to verify a couple of things with SRE before :) [15:52:59] o/ [15:54:25] elukey: im kinda interested in following along on your rocm project for the hackathon a bit (not sure how much i can contribute tho lol) [15:54:51] accraze: definitely! I think that the main purpose could also be to discuss what we do know, how we use it, etc.. [15:54:59] the more knowledge is spread the better [15:55:26] right on! yeah i can there is alot to discuss around the subject [15:55:49] lol can't type yet, need more coffee [17:18:39] aaand now [17:18:39] botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the ListObjects operation: The request signature we calculated does not match the signature you provided. Check your key and signing method. [17:18:53] this is the storage initializer, but I thought I had solved this mess [17:20:53] ah! I found the issue [17:20:56] old image being used [17:24:14] old storage initializer image? [17:28:29] exactly yes [17:28:36] I am about to deploy the new version of the chart [17:29:14] I had to add some ENV values hardcoded in the Docker image, they are fixed in a pull request that I sent (but that will be available only in 0.7 sigh) [17:29:21] (so kserve) [17:55:10] ah I think that I have to create a new docker image [17:55:16] ok will try tomorrow :) [17:55:20] have a good rest of the day folks!