[08:18:50] hello folks [08:19:11] so AWS_CA_BUNDLE seems to work from a local boto python venv that I am using on ml-serve1001 [08:19:29] but then there is a problem of the signature_version, we'd need to use 's3' [08:19:40] and there seems to be no way to pass it to boto via kfserving [08:20:09] the example in https://github.com/kubeflow/kfserving/blob/master/docs/samples/storage/s3/README.md mentions annotations in the s3 k8s secret [08:20:31] that points to https://github.com/kubeflow/kfserving/blob/release-0.6/pkg/credentials/s3/s3_secret.go [08:48:49] That Go file reads secrets etc (`S3Config`) from JSON (or YAML) with the names in the annotations of that struct (`json:"s3AccessKeyIDName,omitempty"` means "Read/write this var from/to 3AccessKeyIDName", but if it's empty, omit the field entirely when writing" [08:49:07] s3AccessKeyIDName* [08:57:45] good morning :) [08:58:04] yes but it reads only some pre-defined vars no? the ones on line 45+ [08:58:56] 39-42 [08:59:30] 46-51 I *think* come from outside the file [08:59:53] They only show up as variables (local to the package `s3` [09:00:11] The file never changes or even fills them [09:01:11] yeah 45+ are used like https://github.com/kubeflow/kfserving/blob/master/docs/samples/storage/s3/README.md [09:01:23] that would be nice to change the signature version [09:01:43] even if I am not 100% sure if thanos-swift supports v4 or not, I am checking [09:03:14] The whole s3 credentials file has no (relevant) mention of "version", so I wouldn't know how [09:07:55] I think we'll need to have either v4 support on swift side (if it doesn't offer it yet, trying to get it) or to send a pull request [10:01:10] 10Lift-Wing, 10drafttopic-modeling, 10Machine-Learning-Team (Active Tasks): Configure revscoring topic deployment pipeline - https://phabricator.wikimedia.org/T287788 (10kevinbazira) a:03kevinbazira [10:01:17] 10Lift-Wing, 10drafttopic-modeling, 10Machine-Learning-Team (Active Tasks): Create blubberfile for revscoring topic model server - https://phabricator.wikimedia.org/T287784 (10kevinbazira) a:03kevinbazira [10:02:07] so after a chat with Effie I was able to verify with s3cmd that thanos-swift supports v4 [10:02:26] in my venv on ml-serve1001 I get [10:02:28] botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the ListBuckets operation: The request signature we calculated does not match the signature you provided. Check your key and signing method. [10:02:34] so it may be some other thing [10:23:07] ahahah so to solve --^ I had to add region="US" to the config [10:23:38] totally crazy but we have s3-region to use [10:34:51] * elukey lunch [10:35:01] will finish the battle later :D [13:47:22] ok so what I think we should do is to add a special ENV variable to the docker image of the storage init [13:47:33] something like [13:47:51] ENV AWS_CA_BUNDLE=/path/to/puppet-ca [13:47:59] that is horrible [13:48:20] but the storage initializer is peculiar, its specs are listed in a k8s config map [13:48:35] not the regular container specs (in which we can specify ENV vars etc..) [13:49:00] this is why I am struggling to test this on the fly with kubectl edit [16:05:22] AWS_CA_BUNDLE=/usr/share/ca-certificates/wikimedia/Puppet_Internal_CA.crt [16:10:46] also if I set verifyssl=0 it still tries to validate ssl [16:10:51] * elukey flips tables [16:17:57] o/ [16:18:41] gfdslklgfrd;gfd;lkngfd;lkogf;lkngfd;lkgfd;lkonjgfd;lkonjgfd;lkonjgfd;lkonjdgf;njkodgflkonj;dgf;njkoiegtr;konjgfds;njkodsgf;njlkofdg;lkonjdgf;lkonjgfd;lknjdgf;lknjgfd [16:18:47] lol [16:18:51] okay just needed to get that out of my system [16:18:58] back to email [16:28:13] created https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/711579 [16:30:01] elukey: nice i think that's probably ok for now [16:31:35] accraze: can't find anything better, I kinda hoped there was a way to say "use the system bundles and/or this is the directory to look into" [16:31:45] but can't find one [17:59:03] ok the patch fixed the TLS issue [17:59:10] then it re-appearted another one, namely [17:59:17] botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the ListBuckets operation: The request signature we calculated does not match the signature you provided. Check your key and signing method. [17:59:59] now this happens, IIUC, since with v4 signatures the default AWS region that boto uses doesn't match with what swift expects, that is "US" in theory [18:00:03] or something like that [18:00:20] I tested it on a python-venv on ml-serve1001, and US worked [18:00:22] BUT [18:00:44] kfserving offers only annotations, that set AWS_REGION in the container [18:00:52] and https://github.com/boto/boto3/issues/2574 [18:00:55] * elukey cries in a corner [18:04:54] going to log off, I hope to find a solution tomorrow, posted to slack's upstream chan, we'll see [21:15:21] :w [21:18:42] 10Lift-Wing, 10artificial-intelligence, 10draftquality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Configure draftquality deployment pipeline - https://phabricator.wikimedia.org/T287787 (10ACraze) [21:18:44] 10Lift-Wing, 10artificial-intelligence, 10draftquality-modeling, 10Machine-Learning-Team (Active Tasks): Create blubberfile for draftquality model server - https://phabricator.wikimedia.org/T287783 (10ACraze) 05Open→03Resolved Blubberfile has been tested & merged, marking this as RESOLVED.