[08:23:59] <wikibugs>	 10GitLab (Auth & Access), 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) Last Friday we've done some troubleshooting and tested a lot of different configurations, thanks @SLyngshede-WMF again! In...
[11:05:31] <wikibugs>	 10GitLab: ignoring extra bitmap file warning when fetching from gitlab - https://phabricator.wikimedia.org/T341712 (10hashar) > Are we running a version in which the bug is solved?  There is a comment from May 2020 (which is from before we have setup Gitlab)  which seems to indicate [[ https://gitlab.com/gitlab-...
[11:36:51] <jbond>	 jelto: i was just looking at the gitlab issue and im seeing strange issues
[11:37:21] <jbond>	 but tl;dr with the current confdig im now getting the following
[11:37:22] <jbond>	 " Your account is pending approval from your GitLab administrator and hence blocked. Please contact your GitLab administrator if you think this is an error. "
[11:37:40] <jbond>	 before the last change though i got an error which said "your email is allready taken"
[11:37:53] <jbond>	 can yuo take a look at the logs and see what we have
[11:38:21] <jbond>	 i.e. what email address does my account have
[11:38:50] * jbond thought that wikimedia.org emails where excempt from auth so gussing i got a temp one? 
[11:40:21] <jelto>	 jbond: this is "expected", because we had some spam issues and new accounts are locked.
[11:40:21] <jelto>	 As mentioned in T320390#9018611 there is still some config active from troubleshooting which uses a different, temporary email (you have temp-email-for-oauth-jbond@gitlab.localhost at the moment).
[11:40:21] <jelto>	 We enabled puppet again on gitlab2002 and idp-test1002 but I'm not sure where this temp-email comes from.
[11:40:22] <stashbot>	 T320390: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390
[11:41:05] <jbond>	 temp email is almost definetly comming from gitlab. 
[11:41:20] <jbond>	 did yuo approve the account (dont if you havn't)
[11:41:37] <jelto>	 no I rejected the account 
[11:41:50] <jbond>	 can yuo delete my entreis
[11:42:00] <jbond>	 i just changed something and im now getti9ng a different error
[11:42:29] <jelto>	 What should be deleted? I deleted your new account jbond1 with the temp email
[11:42:48] <jbond>	 ahh cool are there any other accounts for me?
[11:42:54] <jbond>	 if so can you delete them all
[11:43:17] <jelto>	 only your normal account with wikimedia mail
[11:43:30] <jbond>	 yes please delete that as well 
[11:43:36] <jbond>	 (thats promesing)
[11:44:04] <jelto>	 deleted (we noticed Friday that it takes one or two minutes untils that's done)
[11:44:15] <jbond>	 ok cool ill wait a sec
[11:45:53] <jbond>	 jelto: can yuu check my account now?
[11:46:32] <jelto>	 you have the wikimedia mail again, should I approve?
[11:46:37] <jbond>	 yes :D
[11:46:48] <jelto>	 approved
[11:47:37] <jbond>	 ok im in im just going to try one more thing but i think we have a route forward
[11:48:08] <jelto>	 sounds good! But I still don't fully understand the temp-email thing 
[11:48:46] <jbond>	 i thin k thats a config in the case siode which changes how we send the em,ail addresses which i guess means that gitlab cant find the email
[11:49:00] <jbond>	 at which p;oint git addes some temp addresses
[11:49:12] <jbond>	 removinfg that config allows gitlab to find the email in the correct place
[11:49:27] <jelto>	 okay thanks for the explanation :)
[11:49:34] <jbond>	 i have a meeting in 10 mins but ill sendf an update to the task aftetr that
[11:50:15] <jelto>	 great thanks a lot, ping me any time if you need any deletion/approvals or something else
[11:50:18] <jbond>	 ok can you delete my account again just want to try one more thing
[11:50:36] <jelto>	 deleted
[11:50:42] <jbond>	 great thanks
[11:55:45] <jbond>	 jelto: can you confirm the newest account is also with jbond@w.o and then delete it 
[11:56:15] <jelto>	 yep I can see a new account with jbond@wm.o. I'll delete the account
[11:56:28] <jbond>	 great thanks, last test then i have all the pices :)
[12:36:19] <jbond>	 jelto: ok whats my accxount like like now
[12:37:00] <jelto>	 jbond: jbond@wm.o, looking normal
[12:37:07] <jelto>	 should I approve/delete?
[12:37:13] <jbond>	 yes please
[12:37:27] <jbond>	 ok ill send an uipdate to the phab ticket but i thin k we have a way forward now
[12:37:43] <jelto>	 approve or delete?
[12:37:46] <jelto>	 thank :)!
[12:43:25] <jbond>	 jelto: you can approve that one
[12:43:52] <jbond>	 jelto: can yuo dump the record for that account and my account on production to the task
[12:44:15] <jbond>	 there is some subtile difference between them which prevented my from signing up
[12:44:45] <jbond>	 so we will potentially need to do a manual DB update when we migrate proiduction to OIDC (again ill add omre detail to the task)
[12:52:41] <jelto>	 jbond: approved. I can get the information of both users on prod and replica and paste it to the task
[12:53:09] <jbond>	 yes that would be great thanks, obvioiulsy remove anything sensetive (me email is not sensetive)
[12:57:39] <wikibugs>	 10GitLab (Auth & Access), 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) As requested, @jbond your two users on the replica and production dumped via API (`curl "https://gitlab-replica.wikimedia.o...
[13:10:24] <wikibugs>	 10GitLab (Auth & Access), 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10jbond) I did some more testing today and can confirm that the required config is `cas.authn.oidc.id-token.include-id-token-claims=...
[13:25:02] <jelto>	 jbond: do you think it's enough to delete the "authentication identity" in GitLab for every user? https://docs.gitlab.com/ee/api/users.html#delete-authentication-identity-from-user
[13:31:50] <wikibugs>	 10GitLab (Auth & Access), 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10jbond) > from our side we will need to check if cas.authn.oidc.id-token.include-id-token-claims=true is ok to enable globally or i...
[13:32:10] <jbond>	 jelto: can definetly test it
[13:33:16] <jbond>	 jelto: there may also be some setting that allows multiple identies
[13:33:37] <jbond>	 from tyhe example in the link the user has 
[13:33:38] <jbond>	 "identities": [
[13:33:38] <jbond>	     {"provider": "github", "extern_uid": "2435223452345"},
[13:33:38] <jbond>	     {"provider": "bitbucket", "extern_uid": "john_smith"},
[13:33:38] <jbond>	     {"provider": "google_oauth2", "extern_uid": "8776128412476123468721346"}
[13:33:43] <jbond>	   ],
[13:33:48] <jelto>	 I'll do some tests and research
[13:34:19] <jbond>	 ack i also added a not on the task foir something elses to test but that will require us updating the cas config as well
[13:36:09] <jbond>	 omniauth_auto_link_user = ['cas3', 'openid_connect'] might be worth looking at 
[13:36:35] <jbond>	 fyi i find this usefull for discovering gitlab config https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template
[13:37:17] <wikibugs>	 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - https://phabricator.wikimedia.org/T341468 (10LSobanski)
[13:42:10] <wikibugs>	 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - https://phabricator.wikimedia.org/T341468 (10LSobanski) >>! In T341468#9009202, @jbond wrote: >>  operations/debs > This is also a folder with many repos below it  Thanks for pointing this out, I created {T341991}
[13:48:49] <wikibugs>	 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - https://phabricator.wikimedia.org/T341468 (10BTullis) Did you mean to exclude the following? * `operations/dns` * `operations/docker-images` and its children * `operations/dumps` and its children
[13:54:53] <wikibugs>	 10GitLab (Auth & Access), 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) Thanks for troubleshooting this more! I can confirm existing users have `cas3` in the `identities` section. This leads to a...
[13:55:29] <jelto>	 jbond: just deleting the identities with cas3 does not work, same error. Do you want me to configure send_scope_to_token_endpoint=true on gitlab2002?
[13:57:28] <jbond>	 jelto: im about to go into a meeting but can check after thats finished in about an hour
[13:57:43] <jbond>	 fine by me for you to update it now though or i can ping when im finished
[13:58:09] <jelto>	 I'll disable puppet and update the flag to true on gitlab2002 (gitlab-replica)
[13:58:32] <jbond>	 cheers ill let you know once im done
[14:02:14] <jelto>	 gitlab reconfigured. Feel free to test any time after your meeting. We can also test omniauth_auto_link_user setting after that
[14:17:41] <wikibugs>	 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/debs - https://phabricator.wikimedia.org/T341991 (10Aklapper)
[14:39:41] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: GitLabCIPipelineErrors (tweak thresholds of new alert) - https://phabricator.wikimedia.org/T341927 (10Jelto) @LSobanski thanks for fixing the alert description!  This is a new alert and we have to find the correct thresholds and include...
[15:13:27] <wikibugs>	 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/debs - https://phabricator.wikimedia.org/T341991 (10LSobanski) p:05Triage→03Medium
[15:21:07] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10LSobanski) a:05Jelto→03None
[16:29:40] <wikibugs>	 10GitLab (Project Migration), 10API Platform, 10Anti-Harassment, 10Content-Transform-Team, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10TBurmeister)
[18:29:55] <jbond>	 hi jelto if oyu are still about can you delete my account again.  if not can wait till tomorrow
[20:28:36] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (They Live 🕶️🧟): buildkitd: Require use of the blubber frontend when running on trusted runners. - https://phabricator.wikimedia.org/T329220 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_re...
[20:42:16] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (They Live 🕶️🧟): buildkitd: Require use of the blubber frontend when running on trusted runners. - https://phabricator.wikimedia.org/T329220 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner...
[20:42:32] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (They Live 🕶️🧟): buildkitd: Require use of the blubber frontend when running on trusted runners. - https://phabricator.wikimedia.org/T329220 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner...