[10:25:35] Hello, just a quick query. Has anything changed regarding CAS-SSO in the last week or so? We've got an issue affecting some users of Superset (T328152), specifically its Presto connector. It's most likely related to the upgrade that I rolled out last week, but I just thought I'd check to see if anything about the CAS-SSO authentication might have changed too. Thanks. [10:25:35] T328152: Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 [10:44:13] there were some changes by Simon related to OIDC, but I'd be surprised if this would have any impact on CAS auth [10:44:49] other than that, there wasn't anything, CAS is the same and CAS/prod is also not updated to the latest OpenJDK released over the weekend [10:51:02] can we narrow down when it started to fail? [10:51:16] was the 27th the first day it broke for people? [10:53:40] the only other CAS-related change besides the OIDC work was https://gerrit.wikimedia.org/r/c/operations/puppet/+/860551, which merged on the 24th [11:07:37] Thanks moritzm - I'm working on narrowing it down but it's all a bit anecdotal and vague at the moment. [11:13:28] right now, there's little data, but one thing which is at least the case for the two accounts provided where it fails and works is that [11:13:42] for the failing set the uid and CN are different [11:14:01] while it's the same (sans casing which is irrelevant for LDAP/CAS) for the working set [11:14:27] maybe the new Presto release now operates against a different attribute for impersonation? [11:14:53] maybe try to confirm with a few more users if that observation of failing vs working holds [11:20:36] Thanks. Will do. [12:13:48] My theory is that this issue will go away with new CAS-SSO sessions, so I've asked all affected users to log out (http://idp.wikimedia.org/logout) and try again. [12:25:04] if a user has selected "Remember me" for their session, the max session length is 7 days, so that could add up indeed [13:57:39] moritzm: btullis: i took a quick look at the superset code and it has the following so it shold always prefer the uid [13:57:49] $user_header = 'X-Cas-Uid' [13:58:48] 10SRE-tools, 10Infrastructure-Foundations, 10Machine-Learning-Team, 10Patch-For-Review: httpbb with HTTP POSTs and json payload - https://phabricator.wikimedia.org/T328280 (10isarantopoulos) a:03isarantopoulos [13:58:56] and also $remote_user = 'HTTP_X_CAS_UID' [13:59:15] 10SRE-tools, 10Infrastructure-Foundations, 10Machine-Learning-Team: httpbb doesn't support integers in the POST's body - https://phabricator.wikimedia.org/T328120 (10isarantopoulos) a:03isarantopoulos [14:03:29] 10SRE-tools, 10Infrastructure-Foundations, 10Machine-Learning-Team, 10Patch-For-Review: httpbb with HTTP POSTs and json payload - https://phabricator.wikimedia.org/T328280 (10isarantopoulos) After discussing during the review with @RLazarus we went with the second approach. In the aforementioned patch the... [14:18:31] Thanks jbond. I think I'm getting to the bottom of it now. I think it's caused by permissions within superset itself having changed. It made the SQL Lab tab disappear for people who don't have the `sql_lab` role assigned, but if they used a bookmark they could still see it, just not run queries. CAS-SSO was probably fine all along. [14:20:25] btullis: great thanks for the update [15:33:02] 10SRE-tools, 10Infrastructure-Foundations, 10Machine-Learning-Team: httpbb doesn't support integers in the POST's body - https://phabricator.wikimedia.org/T328120 (10isarantopoulos) @elukey I closed this task since your change has already been merged and deployed. [16:10:55] 10netops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10EChetty) [16:14:14] 10netops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10EChetty) [17:16:30] 10SRE-tools, 10Infrastructure-Foundations, 10Machine-Learning-Team: httpbb doesn't support integers in the POST's body - https://phabricator.wikimedia.org/T328120 (10Aklapper) @isarantopoulos: Hi, this task is still open. If this task is resolved, please set the task status to `resolved`. Thanks a lot! [17:33:12] 10SRE-tools, 10Infrastructure-Foundations, 10Machine-Learning-Team: httpbb doesn't support integers in the POST's body - https://phabricator.wikimedia.org/T328120 (10RLazarus) 05Open→03Resolved [17:35:00] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10dcaro) [17:38:46] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10Andrew) We have a ton of rebalancing to do for each of these switches. The C8 deadline we can meet but can we get two weeks t... [18:34:45] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: cr2-drmrs:xe-0/1/1 stuck optic - https://phabricator.wikimedia.org/T324555 (10RobH) CS0907837: > Support, > > We have three items for remote hands to accomplish for us on this request: > > 1) Please pickup DEL0117661, unpackage it into our ra... [19:15:19] 10netops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10colewhite) [19:33:02] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: cr2-drmrs:xe-0/1/1 stuck optic - https://phabricator.wikimedia.org/T324555 (10RobH) p:05Low→03Medium [19:36:27] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: cr2-drmrs:xe-0/1/1 stuck optic - https://phabricator.wikimedia.org/T324555 (10RobH)