[07:14:55] I'm getting 500 erros on authentication on toosladmin: An unexpected technical problem has occured while handing the requested URI /auth/login/. This is probably temporary and should be fixed soon. Please try again later. [07:14:57] You may be able to get further information in the #wikimedia-cloud channel on the Libera Chat IRC network (https://libera.chat/). [07:14:58] Debug information [07:15:00] If you report this error, please include the details below. [07:15:01] Request ID [07:15:03] 7f8d2b6b8a57480a9a0752181854e5a7 [07:39:29] 500 euros (re @Christophe: I'm getting 500 erros on authentication on toosladmin: An unexpected technical problem has occured while handing the requested U...) [07:40:02] Saw this on the first sight. Later: 500 error (re @cvictorovich: 500 euros) [07:40:34] I fixed the missing "r" 😄 Well you know I wish it was true, I would be a tad richer now. (re @cvictorovich: Saw this on the first sight. Later: 500 error) [13:35:47] Hi i have a problem when i try to register to toolsadmin i have a Internal Server error 500 [13:37:47] my login is Malyzolwikson [13:37:54] !help [13:37:54] If you don't get a response in 15-30 minutes, please email the cloud@ mailing list -- https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication [13:38:02] okay [13:39:47] Guest35: please open a Phabricator task with as much information as you have about what you were trying to do and the error message you got. [14:29:54] Guest35: are you the same person as Christophe above? Just checking if we have one issue or several. [14:39:33] I dont know what you talking about [14:44:18] great, then it's probably not you :) [14:44:42] Did you make a ticket? And if not can you tell me what you were doing when you got the 500? (I just tried logging out and back in but that doesn't really prove much) [14:58:35] ok, I see T434904 and am having a look. May not get you a quick fix on a Saturday though :( [14:58:35] T434904: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904 [15:41:26] I updated the phabricator, I had the issue again just now. So the issue has been there since this morning (10h ago-ish) at least [15:54:53] thanks @Cristophe! Is this your first time logging into toolsadmin or has it worked for you in the past? [15:57:52] It worked before everytime I needed. And I can ssh to toolforge and connect to Toolhub or Phabricator, it seems only Toolsadmin doesn't like me anymore. (re @wmtelegram_bot: thanks @Cristophe! Is this your first time logging into toolsadmin or has it worked for you in the past?) [15:58:02] I'm taking it very very personnaly! 😄 [15:58:49] Thanks, this is all very useful. [15:59:24] My guess would be it is broken for most people and I'm the weird one it works for. [16:01:35] If there is anything I can do to help, or retry logging in just tell me I'll try to do it (if I'm near my computer ^^) [16:02:09] ok! I doubt there will be much progress today, too many other things happening on a weekend. [16:05:43] login works for me FWIW [16:06:09] And me [16:06:34] Does it work if you try logging in from an incognito/private/whatever your browser calls it tab? [16:07:20] the registration errors could be explained by https://gerrit.wikimedia.org/r/plugins/gitiles/labs/striker/+/11aa38a78590b9ca7c27320598badad608b92eae%5E%21/, but not the login errors [16:07:39] unless the registration failing gets the open ldap connection in some weird state? [16:08:56] taavi: do we also have reports of registration issues? I missed those. [16:10:04] andrewbogott: the stack trace you pasted shows '/register/done' as the URL path [16:10:11] Yeah. [16:10:19] As though it's trying to write things to ldap on a login... [16:10:27] * andrewbogott checks logs again [16:10:39] do we have a stack trace for a login failure? [16:14:32] yes, added another stack trace to the task [16:15:05] not finding the user shouldn't 500 though... [16:15:32] that's far enough in the process that it is [16:15:44] is the debug log added for T434684 there? [16:15:45] T434684: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684 [16:18:25] I don't see it... but we know at least one id it's happening for consistently [16:19:54] oh, here's a little bit more of the log, added to task [16:21:23] this could be a paging issue of some sort but then I would it expect to happen to new users, not established ones [16:21:40] also note that striker was redeployed 2 days ago [16:24:11] I bet the failure happens for people w/out an associated SUL account [16:24:48] wait. I might have just realized what's going on [16:25:08] I bet this is python's copy-by-reference biting us (well, me) [16:26:15] oh, is user.ldapuser scrambled by the time we get to check_goal_ssh(user.ldapuser) ? [16:26:45] the dn it's searching for looks right to me... [16:27:08] https://gerrit.wikimedia.org/r/plugins/gitiles/labs/striker/+/11aa38a78590b9ca7c27320598badad608b92eae/striker/labsauth/utils.py#90 only expects to add the object class to the newly created object [16:27:23] but the array is shared between all LdapUser instances [16:27:35] so login fails for users who do not have that object class, and registration fails because of duplicates [16:30:34] I understand how that line is bad but don't quite understand how it's producing the particular failure we see. Why would it make an ldap lookup of a particular dn fail later? [16:30:46] Or is it just that that error message is unreliable? [16:31:18] * andrewbogott happy for taavi to spend time fixing rather than explaining [16:33:15] so, the 'core' issue is that we have a bunch of old users without the wikimediaPerson object class, but striker wants to use fields in that object class for (at least in the current state) newly created users who do have that class [16:33:28] https://gerrit.wikimedia.org/r/c/labs/striker/+/1326016 [16:34:21] basically that append() makes it so that once a single user has been registered, that striker runtime will stop being able to handle LdapUser objects without the object class, because the same attribute is used for ldap search filters and writing new objects to the tree [16:34:42] that patch is untested but i'm 95% confident it fixes the issue [16:34:45] oh, the search filter! OK, that's the part I was missing. [16:35:46] and I didn't initially connect the issues with existing users using striker to the patch because it should have only affected new account creations [16:36:20] So... u.object_classes falls through to a shared list unless u contains an object_classes member and then it doesn't? [16:36:30] * andrewbogott glares at python [16:37:17] oof. this is one of the reasons I started disliking python (and eventually stop pretty much using it) [16:37:52] taavi, if you need to go participate in real life I can merge and deploy [16:38:04] * andrewbogott has 45 minutes or so before real life [16:38:59] andrewbogott: please do, thanks [16:40:43] * andrewbogott mashes the +2 button [16:40:44] currently object_classes points to a list object shared between all instances. my patch updates the constructor to replace that reference with one to a new list object for that instance only with the exact same elements, so that it can be modified without affecting other instances [16:41:28] perryprog: python is nice enough that you end up forgetting stupidity like this until it's too late [16:41:33] exactly! [16:41:42] yep, I follow, I'm just surprised about the implicit shared member thing. [16:42:49] but might not be if I read more of the code [16:43:01] that being said, Django is ihmo the best thing out there at the moment for a basic web-based database crud thing. doesn't mean I like Django or Python, I just don't know of any better alternatives [16:47:35] I think you could argue something like rails, but honestly I do agree [16:48:43] the rails community, uh, does not exactly seem like one that i want to associate myself with at the moment [16:49:46] sooo true [16:53:42] i used to like laravel but never really got used to how most of that community tends to prioritize code "beautifulness" (if that's how you can say it) over performance and other factors. plus so many of the projects in that space ended up being way more commercial than i was comfortable relying on and contributing to [16:54:53] I mean if I could pick without regard to, uh, "contributor approachability" I'd probably go for something in Clojure [16:57:08] !log admin optimistically deploying a blind fix for striker issue T434684 [16:57:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:57:16] T434684: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684 [17:01:54] @Christophe and Guest35, can you try now? [17:03:14] I didn't think about that, and actually it does. Let me update the phab ticket (re @wmtelegram_bot: Does it work if you try logging in from an incognito/private/whatever your browser calls it tab?) [17:03:24] it works [17:03:30] Well, that's prolly because of the fix that was just deployed [17:04:18] yeah, likely unrelated to the private browser but can you test in your normal browser session as well just to be sure? [17:05:55] I did both and both worked (re @wmtelegram_bot: yeah, likely unrelated to the private browser but can you test in your normal browser session as well just to be ...) [17:06:17] great, thank you for reporting and testing [17:06:55] That's the least I could do, and thank you for fixing that on a Saturday!!! [17:07:08] I hope you will get to enjoy the reste of your week-end 🙂 [17:09:10] thx taavi [17:10:12] Thanks taavi too! [23:48:47] Hello, I had a quick question. I'm working on a tool on Toolforge. [23:48:48] Is there guidance/a recommendation on storing long-term secrets for Toolforge tools? I know there's an environment variables service to store secrets/config info. [23:48:48] I'm not sure if those environment variables are persisted long-term though. IE: if there's a server restart/update/etc. - are those environment variables saved? Or are they only stored in memory? [23:48:49] For context I'm coming from using GitHub Actions/Azure DevOps where I'm used to storing my secrets inside the CI platform as a secret. I checked and I don't think Gitlab secrets are enabled currently. [23:48:49] Thanks very much.