[08:52:39] I'd be very glad of another pair of eyes from this group on this patch, please: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1344034 [08:52:39] It is about implementing this: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery [08:52:44] Thanks. [09:02:33] btullis: may I ask the obvious question of "why can't RADOS authenticate itself?" [09:25:56] jayme: Yes, of course. This is part of a project to migrate away from Hadoop and HDFS to Kubernetes with Ceph/S3. Top-level epic here: T435461 - Most relevant section of the security model doc here: https://docs.google.com/document/d/1XMSv9Sc9IWs4GXNCZeCyvO0R8l5ej1_H9zBv_poog9g/edit?tab=t.0#heading=h.2k9eri5uc6cm [09:25:57] T435461: [Epic] Migrate the Data Lake from Hadoop to Kubernetes and Ceph/S3 - https://phabricator.wikimedia.org/T435461 [09:28:36] At the moment, we use Kerberos to provide short-term credentials for accessing HDFS, YARN, the Hive metastore, and Presto. But in the new model Kerberos is replaced with the Secure Token Service (STS) https://docs.ceph.com/en/latest/radosgw/STS/ [09:29:28] If you are a person and you want to access the files on the new S3 data lake, you will authenticate against the CAS-SSO system, backed by LDAP and Wikimedia Developer Accounts. [09:33:12] However, if you are a Spark job or an Airflow task, or anything else running under Kubernetes, then you will use a projected service account token (https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#launch-a-pod-using-service-account-token-projection) that the Ceph Rados Gateway will be able to use to validate the request. [09:34:46] Once validated, either by CAS-SSO, or by Kubernetes, then an IAM policy (https://docs.ceph.com/en/latest/radosgw/iam/) will be applied, permitting access to the right buckets and verbs etc. [09:37:07] This ticket has a summary of the authentication parts, too: T435470 [09:37:08] T435470: Enable OIDC authentication and IAM authorization on the Data Platform Ceph clusters - https://phabricator.wikimedia.org/T435470 [09:52:33] I bet you wished you hadn't asked, now :-) [09:53:51] not sure I put this right. I meant: Why do you need unauthenticated access to those endpoints? Can't you just provice the RGW with credentials to access the endpoints? [10:01:28] Oh, I see. I'll have a think. One sec. [10:02:44] (I thought you meant, why doesn't RADOS just keep a list of users, which is what it does today.) [10:53:17] I went down a bit of a rabbit-hole, but I think it wouldn't work and I don't really think that it is supposed to, anyway. [10:57:15] Here is where the RADOS Gateway does the HTTP call to get `.well-known/openid-configuration` https://github.com/ceph/ceph/blob/squid/src/rgw/rgw_rest_sts.cc#L300 - It doesn't currently have the capability to use a client cert and key for mutual TLS. So although they ultimately use the same PKI for both the Rados Gateway and the K8S API, it's not possible to get this lookup to use authentication without a patch to Ceph. [10:57:44] alright then...I was just curious :) [11:24:08] Thanks for inquiring. It makes me go back and check my assumptions I think that the main thing is that OIDC was designed to a relying party to fetch these without credentials. That section I first linked to says: [11:24:14] > Administrators may, additionally, choose to bind the role to system:authenticated or system:unauthenticated depending on their security requirements and which external systems they intend to federate with. [11:29:41] I then went off to check the OIDC spec and the RFCs, but they don't /require/ that these well-known URIs are accessible without unauthentication. But that is how they all seem to work. We allow https://idp.wikimedia.org/oidc/.well-known/openid-configuration without auth. We're not going to be using a public URL and we're still firewalling access to the API, so it's still pretty tightly controlled.