[13:35:40] <andrewbogott>	 slyngs: two easy questions. 1) Is there anything special I need to do to get this change deployed? and 2) does the regex look right to you? I want to support redirects to both https://openstack.eqiad1.wikimedia.cloud:25000/protected and also https://keystone.eqiad1.wikimedia.cloud/protected (with no port specified for the latter)  https://gerrit.wikimedia.org/r/c/operations/puppet/+/1125249/4/hieradata/role/common/idp.yaml
[13:36:53] <andrewbogott>	 um, sorry,those urls should be https://openstack.eqiad1.wikimediacloud.org:25000/protected and   https://keystone.openstack.eqiad1.wikimediacloud.org/protected
[13:37:02] <andrewbogott>	 (one with a port specified an one without)
[13:37:02] <slyngs>	 1) Should deploy automatically when merged. 2) Let me just test, it looks about right, but I'll just check
[13:38:01] <slyngs>	 Regex seems fine as well
[13:38:21] <andrewbogott>	 cool, in that case I have a third question :)
[13:39:00] <andrewbogott>	 If you navigate to https://labtesthorizon.wikimedia.org/ you will see that there's a mismatch
[13:39:23] <slyngs>	 I have seen that it sometimes doesn't work when you change an existing service
[13:39:36] <andrewbogott>	 labtest is from a different patch, this one: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1125264
[13:39:50] <slyngs>	 What we can do it try to do a quick failover to the other host, after a restart
[13:39:55] <andrewbogott>	 hang on
[13:40:04] <andrewbogott>	 so, I'm being confusing, talking about two different services.
[13:40:09] <andrewbogott>	 I think the prod change is working properly
[13:40:30] <andrewbogott>	 but what's not working is the change to cloudidp-dev.wikimedia.org
[13:41:20] <andrewbogott>	 If you think it's a caching issue I can just reboot the host that cloudidp-dev is running on since it's not a prod service.
[13:42:39] <slyngs>	 Cool, let's try that first, again I have seen service changes not being picked up
[13:43:26] <andrewbogott>	 ok, rebooting...
[13:44:40] <slyngs>	 Oh, the regex'es aren't the same
[13:45:15] <slyngs>	 https://(keystone.)?.openstack.*.wikimediacloud.org.*
[13:45:15] <slyngs>	 https://(keystone.)?openstack.*.wikimediacloud.org.*
[13:45:29] <andrewbogott>	 oh! I see it! an extra 
[13:45:30] <andrewbogott>	 .
[13:45:41] <slyngs>	 Yup :-)
[13:47:27] <andrewbogott>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1125428
[13:47:38] <andrewbogott>	 that should help!
[13:48:05] <slyngs>	 +1
[13:50:59] <andrewbogott>	 ok, that fixed it
[13:51:06] <andrewbogott>	 which means now I can start working on the /actual/ problem :(
[13:51:22] <slyngs>	 What's the actual problem?
[13:51:32] <andrewbogott>	 I'll only implicate you in that one if you have a couple hours to burn :)
[13:51:58] <slyngs>	 Sadly I one have about 10 - 15 minutes :-)
[13:51:58] <andrewbogott>	 The login cycle works on my hotspot network but not on my wifi network. And other users are reporting the same, that it doesn't work on certain home networks.
[13:52:09] <andrewbogott>	 I thought it was the port 25000 thing, so all of the above was about moving things to 443
[13:52:13] <andrewbogott>	 but it still breaks for me on 443
[13:52:34] <slyngs>	 Oh that is weird
[13:52:47] <andrewbogott>	 oh, actually...
[13:52:49] <andrewbogott>	 I'm looking at https://phabricator.wikimedia.org/T388137
[13:53:05] <andrewbogott>	 but that's different from what I get, the message I see is 
[13:53:07] <andrewbogott>	 https://www.irccloud.com/pastebin/SKMIZOFN/
[13:53:41] <slyngs>	 What was the request?
[13:54:38] * andrewbogott trying to figure out how to cut and paste from web tools...
[13:55:25] <andrewbogott>	 https://usercontent.irccloud-cdn.com/file/AQt8juFp/Screenshot%202025-03-07%20at%209.54.50%E2%80%AFAM.png
[13:55:34] <andrewbogott>	 that probably doesn't show the good bit
[13:56:10] <slyngs>	 Probably in the "Request" tab
[13:58:18] <andrewbogott>	 https://usercontent.irccloud-cdn.com/file/TLPnEapY/Screenshot%202025-03-07%20at%209.57.58%E2%80%AFAM.png
[14:00:01] <slyngs>	 That doesn't seem particularly broken.
[14:01:06] <andrewbogott>	 yeah
[14:04:06] <andrewbogott>	 If the network I'm behind is blocked someplace it wouldn't produce a 400 would it?
[14:05:03] <andrewbogott>	 going to try with a working network and see if I can tell the difference
[14:05:52] <slyngs>	 Nope I don't think so,... Given that you're seeing it on a hotspot, I do wonder if it's some CGNAT in play.
[14:07:18] <andrewbogott>	 actually it works on the hotspot
[14:07:22] <andrewbogott>	 and fails with the home fiber network
[14:07:37] <slyngs>	 Aah... that's worse somehow
[14:16:44] <slyngs>	 andrewbogott: I have to run, but drop me a message if there's anything you need me to look at.
[14:16:59] <andrewbogott>	 ok! You've already gotten me past one typo, thanks for that
[14:35:42] <andrewbogott>	 slyngs (when you're back), here is the failure in my apache logs:
[14:35:45] <andrewbogott>	 https://www.irccloud.com/pastebin/LDn2793o/
[14:36:09] <andrewbogott>	 Can't imagine why a change in network would scramble my cookie
[16:37:23] <andrewbogott>	 dhinus: I have a scheme for (maybe) fixing that horizon/401 issue; are you wrapping up for the day or are you interested in having a look?
[16:37:44] <andrewbogott>	 oddly unrelated to recent convo with simon which, that change didn't do much
[17:06:14] <dhinus>	 andrewbogott: I'm still around and I'm interested :)
[17:07:02] <dhinus>	 I was trying to think of what could explain it but it looks more and more mysterious
[17:09:21] <andrewbogott>	 I checked and the mapping between idp and keystone uses the name field. I'm thinking if we map via id instead of name then there's no chance to screw up encodings since IDs are ascii.
[17:10:03] <andrewbogott>	 right now the mapping looks like this:
[17:10:24] <andrewbogott>	  [{'local': [{'user': {'name': '{0}', 'type': 'local', 'domain': {'name': 'Default'}}}] | 'remote': [{'type': 'HTTP_OIDC_NAME'}]}]
[17:10:52] <andrewbogott>	 just switching name to id and OIDC_NAME to OIDC_ID doesn't work, but that whole HTTP_OIDC_NAME thing is kind of a black box to me.
[17:11:18] <andrewbogott>	 If we can figure out what the data structure looks like that oid is providing to keystone then we'll know what to call it, probably
[17:15:40] <dhinus>	 hmmm I wonder how much of this is hardcoded into keystone, but we can always check the source
[17:16:12] <dhinus>	 my brain is a bit fried on a Friday eve but this looks promising
[17:16:17] <andrewbogott>	 I'm kind of assuming that the encoding is hardcoded, hence wanting to work around it
[17:16:34] <dhinus>	 ack
[17:17:41] <dhinus>	 I'm more puzzled by your issue where it only works on one network
[17:17:57] <dhinus>	 at least here we know there's a wrong encoding somewhere and we "just" have to find where, or how to work around it
[17:17:58] <andrewbogott>	 yeah, trying to set that one aside for the moment since I already tried the one thing that I thought would work
[17:18:19] <dhinus>	 agreed we can try fixing the encoding one first, so at least we get that one out of the way
[17:18:27] <andrewbogott>	 I was thinking that if you're handier with web tools (and have two working eyes) maybe you could inspect the login dance and see if you can find the blob of data that idp is passing to keystone and/or horizon after you auth with the idp popup?
[17:18:37] <dhinus>	 let me try
[17:18:47] <andrewbogott>	 maybe with labtesthorizon since that's easier to hack
[17:20:11] <dhinus>	 I'm now getting 401 on labtesthorizon, which I guess is due to your temp changes?
[17:21:35] <andrewbogott>	 yes, but that's after the mapping happens...
[17:21:40] <andrewbogott>	 (on the phone now sorry)
[17:21:47] <dhinus>	 yes I'm inspecting the network calls
[17:21:50] <andrewbogott>	 cool
[17:24:29] <dhinus>	 I see a POST call to the keystone endpoint with a big "id_token" in the paylod, but that's encoded
[17:24:54] <dhinus>	 maybe it's just base64
[17:27:53] <dhinus>	 yep it's 3 base64 strings separated by "."
[17:29:35] <dhinus>	 the last one is a signature
[17:29:49] <dhinus>	 I'll send you the decoded payload in a dm
[17:29:51] <dhinus>	 andrewbogott: 
[17:29:52] <andrewbogott>	 can  you tell what fields?
[17:34:42] <dhinus>	 looks like the id is in the field preferred_username
[17:37:59] <dhinus>	 retried now, still 401
[17:38:28] <dhinus>	 we probably have to dig deeper into the keystone code :/
[17:40:24] <andrewbogott>	 dhinus: can you try once more? I'm not seeing a failed lookup in the logs...
[17:40:32] <andrewbogott>	 although I guess if the mapping is bogus then we wouldn't get that far
[17:41:27] <dhinus>	 retried, still 401
[17:41:43] * dhinus reads https://docs.openstack.org/keystone/latest/admin/federation/mapping_combinations.html
[17:42:22] <andrewbogott>	 yeah, doesn't appear in the logs which makes me think it's choking on that mapping
[17:45:22] <andrewbogott>	 I turned on debug logs, will you try once more?
[17:45:28] <dhinus>	 sure
[17:45:42] <andrewbogott>	 (I can't try because of the network thing and also because my username = my id so I get fake success sometimes)
[17:45:51] <dhinus>	 401
[17:51:46] <andrewbogott>	 try again?
[17:52:45] <dhinus>	 401 :/
[17:56:12] <andrewbogott>	 ok, time for some debug lines
[17:58:40] <dhinus>	 there are some notes here https://github.com/openstack/keystone/blob/master/doc/source/admin/federation/openidc.inc#configure-mod_auth_openidc
[17:58:46] <andrewbogott>	 logs are saying 'remote_id_parameter = HTTP_OIDC_ISS' which I don't entirely follow but maybe remote_id_parameter isn't what I think it is
[17:59:17] <andrewbogott>	 can you try another failed login?
[17:59:37] <andrewbogott>	 oh, the vhost, hm...
[18:00:14] <dhinus>	 ISS apparently stands for "issuer"
[18:00:33] <andrewbogott>	 seems like
[18:00:34] <andrewbogott>	 OIDCScope "openid email profile"
[18:00:41] <andrewbogott>	 should include the field we want? But name already isn't in there
[18:01:00] <andrewbogott>	 try another login?
[18:01:03] <dhinus>	 I was also thinking about it, but it's already in the payload so I'm not sure if adding it there would make a difference
[18:01:13] <dhinus>	 trying a login
[18:01:26] <dhinus>	 401
[18:03:29] <andrewbogott>	 hm
[18:03:32] <andrewbogott>	 ANDREW, complete idp content is {'id': 'openid', 'domain_id': '3adf532eb4e14cd79dcdcdcb22e096a2', 'enabled': True, 'description': None, 'remote_ids': ['https://cloudidp-dev.wikimedia.org/oidc'], 'authorization_ttl': None}
[18:03:38] <andrewbogott>	 absolutely not what i was looking for
[18:05:00] <andrewbogott>	 I'm reading keystone/federation/utils.py
[18:05:10] <andrewbogott>	 or at least squinting at it
[18:12:31] <andrewbogott>	 dhinus: log in again?
[18:12:38] <dhinus>	 trying
[18:12:41] <andrewbogott>	 I didn't change anything but debug lines
[18:13:00] <dhinus>	 401
[18:15:48] <dhinus>	 OIDCScope is correct: scopes are groups of claims, and the "profile" scope contains the "preferred_username" field
[18:17:19] <dhinus>	 I have to log off... I can do a couple more attempts later if it helps!
[18:17:50] <andrewbogott>	 ok. Thank you for helping!
[18:18:11] <dhinus>	 you're welcome, thanks for looking into this!
[18:18:17] <andrewbogott>	 omg I swapped a - with a _
[18:18:24] <andrewbogott>	 can you stick around for another 3 minutes while I fix that?
[18:21:21] <dhinus>	 sure
[18:21:32] <andrewbogott>	 try now?
[18:21:57] <dhinus>	 IT WORKED
[18:22:02] <dhinus>	 amazing
[18:22:10] <andrewbogott>	 the field was OIDC-preferred_username
[18:22:14] <dhinus>	 the unexpected happy end
[18:22:16] <andrewbogott>	 mixed - and _
[18:22:26] <andrewbogott>	 ok I'll try rolling this out in eqiad1 and see what it breaks :)
[18:22:29] <andrewbogott>	 thank you again
[18:22:35] <dhinus>	 thank you!
[18:22:40] <dhinus>	 have a good weekend :)
[18:24:08] <andrewbogott>	 you too
[18:47:27] <andrewbogott>	 Now I need to sit in the dark for a while. See y'all on the 18th if nothing explodes before then