[08:48:25] oh my, just realized that the toolhub<->toolforge relationship can be *-* (one tool in toolforge can define many toolhub entries and vice-versa), I though it was 1-1 or 1-0 if no toolforge tool matched, things are more complex than they seemed! [08:48:57] I wonder, is there any info on toolhub that directly relates to the toolforge tool? (something like `toolforge_ids` or similar) [09:09:16] dcaro: hmm, that's what I thought too. what is the purpose of mapping one-to-many and where did you discover that? [09:53:14] heads up, I'm deploying again the refactored maintain-kubeusers to toolsbeta [10:21:33] blancadesal: so there's a task in which I requested the source code for the tool, but it turns out it was already public, just as two different tools in toolhub (https://heritage.toolforge.org/toolinfo.json has two entries heritage-api and heritage-daily-uploads) [10:21:56] (it was to help with the migration) [10:24:17] arturo: do you want any assistance/help? [10:24:55] dcaro: at the moment I'm trying to debug why the code can't detect a disabled account [10:25:33] I'm crafting a little tester script, because I suspect the LDAP library is not doing what I want it to do [10:29:08] ok, theory is proven correct, the LDAP query is not fetching the required data to know if an account is disabled [10:29:11] https://www.irccloud.com/pastebin/zYnHytAN/ [10:29:44] how was this working before the refactor? [10:29:56] good question [10:30:06] I guess it wasnt working :-( [10:30:36] andrew mentioned about hidden attributes [10:31:41] * arturo asks gemini [10:32:31] * arturo gets `I'm a text-based AI and can't assist with that.` :-( [10:38:04] ok, I think I found the problem [10:41:15] * arturo creates T366263 [10:57:01] I'm not sure how maintain-kubeusers should handle these double accounts [10:57:10] or double entries in LDAP [10:58:14] for now I think I will delete the extra account and leave the disabled one only [10:58:27] is there any other double account? [10:58:40] (are there any double accounts in tools?) [10:58:49] I have no idea [11:02:28] maybe it's just one account that was duplicated by mistake? (the first entry is from 2021, the second entry from 2024) [11:03:20] yesterday I played with that account in LDAP, and andrew too. The second entry could have been created yesterday [11:03:28] (by mistake) [11:03:37] accounts on tools.* seem to use `uid=tools.*` instead of `cn=tools.*` [11:03:58] about the other inconsistencies that you detected, I don't know about them [11:04:19] maybe we were using `cn=` before and changed to `uid=` at some point? [11:04:45] I have no idea :-( but I have never heard of such change [11:04:53] I suspect this has been manually mangled [11:05:22] I tend to fat-finger all these things, so maybe I created the inconsistencies myself over the years [11:06:07] hmm, the docs for toolsbeta creation say `cn=` https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta [11:06:59] I believe I created those docs, so if people have been following the wrong docs over the year... then that explains the current inconsistencies [11:07:42] * arturo brb [11:07:46] striker user uid https://gerrit.wikimedia.org/r/plugins/gitiles/labs/striker/+/refs/heads/master/striker/labsauth/models.py#217 [11:07:51] (I think) [11:08:29] so probable that's it :) [11:08:34] *probably [11:08:38] should be easy to fix I guess? [11:11:46] * dcaro lunch [11:53:08] wait, one is the posixgroup (dn: cn=)and the other is the actual possixAccount (dn: uid=) [11:55:07] hmm, toolsbeta.test8 is the only one that shows both in the search [11:55:08] https://www.irccloud.com/pastebin/TrGZnMDD/ [11:56:54] yeah, so this is probably the only entry that's wrong [11:56:56] when you search using the `cn=` then they showup [11:57:03] and that's probably me yesterday messing with it [11:58:10] interesting, when searching with `cn=toolsbeta.test8` then you get 3 entries [11:58:30] one cn= should exists for real, is the tool posixGroup [11:59:11] and it should have `one with `structuralObjectClass: groupOfNames` not `structuralObjectClass: person` (test8 has both, the two `cn` entries one with each) [11:59:25] I think I messed up the quotes xd [12:00:04] maybe we should delete test8 entirely and create a new fresh one [12:00:47] I think we can just delete the entry with `dn: cn=toolsbeta.test8,ou=people,ou=servicegroups,dc=wikimedia,dc=org`, note that it has both `ou=people` and `ou=servicegroups` [12:00:54] ok [12:01:41] * arturo hope there are backups [12:02:08] moritzm: are you available to assist with some LDAP things? [12:03:14] this is the command I would like to run [12:03:16] sudo ldapdelete -x -h serpens.wikimedia.org -p 389 -D "cn=admin,dc=wikimedia,dc=org" -W cn=toolsbeta.test8,ou=people,ou=servicegroups,dc=wikimedia,dc=org [12:12:51] this seems to only match the one we want to delete `root@mwmaint1002:~# ldapsearch -x '(&(cn:dn:=toolsbeta.test8)(!(uid:dn:=toolsbeta.test8)))'` [12:13:54] (/me fooling around with ldap) [12:14:21] I think I can wait for andrew to be awake to confirm this is the right thing to do [12:15:40] sure [13:54:05] * andrewbogott is partly awake but hasn't read the backscroll yet [13:57:23] I may be missing something -- this just one of many test tools in toolsbeta, so pretty low stakes if it goes wrong isn't it? [14:02:34] unless it deletes something not in beta xd (shared ldap risk) [14:03:25] I would expect each tool to have two entries: one groupofnames which tracks the members of a tool, and one person, which is the tool service user [14:03:30] That's what I see for test8 [14:04:04] for instance in the disable-tool code we have the following two definitions: [14:04:06] tool_dn = "cn=%s.%s,ou=servicegroups,dc=wikimedia,dc=org" [14:04:19] tool_user_dn = "uid=%s.%s,ou=people,ou=servicegroups,dc=wikimedia,dc=org" [14:05:05] arturo: does that explain what you're seeing? [14:07:34] test8 has 3 entries, not 2 [14:08:16] one with "cn=%s.%s,ou=servicegroups,dc=wikimedia,dc=org", one with "uid=%s.%s,ou=people,ou=servicegroups,dc=wikimedia,dc=org", and one with "cn=%s.%s,ou=servicegroups,ou=people,dc=wikimedia,dc=org" [14:08:26] that third one is the one we think should not be there [14:09:24] * andrewbogott looks again [14:09:52] ah, so it does :( [14:15:05] so everything under ou=servicegroups,ou=people,dc=wikimedia,dc=org is likely in error... [14:15:11] * andrewbogott fights with ldapsearch to find those [14:15:56] not really, users have that in their dn [14:16:01] "uid=%s.%s,ou=people,ou=servicegroups,dc=wikimedia,dc=org" [14:16:19] anything without uid in the dn and with ou=servicegroups,ou=people,dc=wikimedia,dc=org might be what we are looking for? [14:16:54] My eyes may deceive me but I think in one case you pasted an entry with ou=servicegroups,ou=people and in the other case ou=people,ou=servicegroups [14:17:28] It's not ever correct to have a servicegroups entry /under/ people is it? [14:17:44] oops, yes sorry, it's the other way around (manually edited it) [14:18:06] ok, that explains why my search didn't find anything [14:18:12] I think all we need is the command I pasted above (and in phab) [14:19:25] that should sort the only account we have found so far yes [14:22:58] hmmm... I don't understand ldapsearch syntax xd, this still gives me both entries [14:23:01] https://www.irccloud.com/pastebin/Gf70z4nd/ [14:23:13] curious, I can see that third entry with ldapsearch but not in ldapvi [14:27:29] I think ldapvi is precisely what got me confused into creating the additional (wrong) entry [14:27:48] ok, so if no further comments, I'll run the aforementioned command [14:28:01] sudo ldapdelete -x -h serpens.wikimedia.org -p 389 -D "cn=admin,dc=wikimedia,dc=org" -W cn=toolsbeta.test8,ou=people,ou=servicegroups,dc=wikimedia,dc=org [14:28:31] arturo: that looks correct to me [14:28:47] although I'd like to see a query that finds all examples rather than just that one [14:28:53] ldap_delete: No such object (32) [14:28:55] :-( [14:30:43] hmm.... it seems that matching only by the dn is not that easy: https://superuser.com/questions/1652940/ldap-filter-to-search-for-a-dn [14:31:56] and given that both entries have both attributes in them (cn and uid), trying to filter by one of the other gives both (`(cn:dn:=toolsbeta.test8)` and `(uid:dn:=toolsbeta.test8)` end up looking in the object after looking in the `dn`) [14:32:13] But I think you can do -b cn=toolsbeta.test8,ou=people,ou=servicegroups,dc=wikimedia,dc=org [14:32:31] and then wildcard with abandon [14:32:55] ldapdelete: unrecognized option -b [14:33:03] dammit [14:33:08] well at least there's -n for testing [14:33:21] ldapsearch works with the -b you passed :) [14:33:30] cursed LDAP [14:33:31] ldapdelete uses the full dn only no? (by the help page it seems) [14:38:29] lol I must be using a different version of ldapdelete because mine doesn't seem to take a -h [14:38:50] are you in mwmaint1002 ? [14:38:56] nope, a cloudcontrol [14:39:26] I'm on mwmaint1002, let me know if you think I should go to cloudcontrol instead [14:39:32] nope, either should work [14:39:45] and now I've caught up with you and am getting No such object (32) [14:40:52] do we have a query that returns the wrong object? [14:41:04] well, I guess the one I pasted on phab [14:41:06] ldapsearch -x -b cn=toolsbeta.test8,ou=people,ou=servicegroups,dc=wikimedia,dc=org [14:41:09] also that [14:41:30] can we hit serpens with that query to see if it's there too? [14:42:41] oh, good idea [14:42:59] it isn't [14:43:04] https://www.irccloud.com/pastebin/cojOQNFj/ [14:43:46] that's the right one only though, with `dn: uid..` [14:43:51] so maybe we need to run the delete command not against serpens [14:43:55] let me try this: [14:44:12] sudo ldapdelete -x -D "cn=admin,dc=wikimedia,dc=org" -W -n cn=toolsbeta.test8,ou=people,ou=servicegroups,dc=wikimedia,dc=org [14:44:34] * andrewbogott delighted by this latest wrinkle [14:44:57] https://www.irccloud.com/pastebin/hlf63DjK/ [14:45:00] arturo: ideally that will tell you not allowed because it's a r/o host. But please try it! [14:45:00] server is RO [14:45:03] yeah, ok [14:45:11] interesting.... how would this happen? [14:45:24] have you checked to see if hat phantom cn= record appears for every tool? [14:45:27] so I wonder: where is the entry stored? [14:47:43] (my question about every tool is based on the idea that this is an illusion produced by ldapsearch) [14:48:00] no, I think it is a real thing [14:48:08] ok [14:48:09] 🪄 [14:48:18] the coded I added in maintain-kubeusers to detect such case already works fine [14:48:40] 2024-05-30T14:34:57.205056143Z duplicated account detected in LDAP, entry for 'test8'. Check LDAP tree. [14:48:40] 2024-05-30T14:34:57.205246519Z duplicated account, version A: name: 'test8' uid: '54867' data_dir: '/data/project/test8' is_admin: 'False' namespace: 'tool-test8' is_disabled: 'True' configmap_name: 'maintain-kubeusers-test8' [14:48:40] 2024-05-30T14:34:57.205325363Z duplicated account, version B: name: 'test8' uid: '54867' data_dir: '/data/project/test8' is_admin: 'False' namespace: 'tool-test8' is_disabled: 'False' configmap_name: 'maintain-kubeusers-test8' [14:48:56] (note the is_disabled diff) [14:49:21] yeah, ok [14:50:11] seems it's not on seaborgium either [14:51:13] is there a way I can make full search for users? like `ldapsearch -x -b ou=people,ou=servicegroups,dc=wikimedia,dc=org | grep dn | grep tools | grep cn=` that should give any of those weird entries [14:51:16] blancadesal: "what is the purpose of mapping one-to-many" -- the purpose is that the toolingo.json standard is about what the end user uses and not how things are hosted. There are many, many Toolforge accounts that use one webservice to host a pile of different tools. An easy example is https://hay.toolforge.org/ [14:51:22] (dn: cn=toolsbeta.fourohfour,ou=people,ou=servicegroups,dc=wikimedia,dc=org is another one it seems) [14:51:56] dcaro: you might run into paging with a search as big as that [14:52:05] bd808: about that, toolsadmin is not able to handle multiple tool toolinfos right? [14:52:57] no, it handles that use case [14:53:14] andrewbogott: what is the LDAP server in codfw? [14:53:19] maybe is from codfw? [14:53:52] dcaro: looks for the "add toolinfo" button at the bottom right of any screen like https://toolsadmin.wikimedia.org/tools/id/admin [14:54:25] just saw! nice! I though that was for creating it from scratch xd [14:54:30] https://usercontent.irccloud-cdn.com/file/9Thom7FD/image.png [14:54:31] arturo: seaborgium which I also checked [14:54:50] but there's also ldaprw1001 and ldaprw2001 in site.pp [14:55:37] I need 5 minutes to eat breakfast before the meeting but will definitely return to this fascinating triviality! [14:56:14] xd [14:56:28] dcaro: if you add `-E pr=5000/noprompt` to your `ldapsearch` query you may have better luck with large queries. That is adding paged result support and requesting 5000 hits per page. [14:56:46] nice, that should be enough for tools right? [14:57:13] I use `ldapsearch -xLLL -P 3 -E pr=5000/noprompt -o ldif-wrap=no -b"dc=wikimedia,dc=org"` with an alias of `ldap` for convenience [14:57:42] that worked! [14:57:51] tools with weird entries: [14:57:53] https://www.irccloud.com/pastebin/zlZlIJOg/ [14:58:00] the 5k page size is quite often large enough to get all in one page, but the magic of that flag is also that it will automatically page through until all the results are read [14:58:00] all in toolsbeta [14:58:12] that's the noprompt? [14:58:20] yeah [14:59:04] what a mess [14:59:26] it was probably harmless until you went looking for it :p [15:00:06] * arturo can't help think about what would be the 2024 replacement of LDAP [15:00:12] bd808: ok, I see [15:04:28] dcaro: what is wrong with those service accounts for the toolsbeta tools? [15:05:18] `ldap "cn=toolsbeta.*" -b "ou=people,ou=servicegroups,dc=wikimedia,dc=org"` cn -- gives me 13 hits for toolsbeta tools in the directory [15:05:48] using "cn=tools.*" gives 3293 hits [15:18:27] bd808: they should be "cn=%s.%s,ou=servicegroups,dc=wikimedia,dc=org" (for a group) or "uid=%s.%s,ou=people,ou=servicegroups,dc=wikimedia,dc=org" (for an account) but have "cn=%s.%s,ou=servicegroups,ou=people,dc=wikimedia,dc=org" (cn + servicgroup + people) [15:21:48] I guess I am confused. `ldap -b "ou=servicegroups,ou=people,dc=wikimedia,dc=org" cn` does not find anything for me. The entries I see in that paste you shared are under "ou=people,ou=servicegroups,dc=wikimedia,dc=org" as I would expect. [15:25:28] oh! cn= vs uid=. I see it now. sorry for being dense :) [15:33:22] yeah, it is a very stupid and mostly cosmetic diff, but I also had to update maintain-kubeusers to handle it [15:35:43] I think all of us took some time to see the issue too xd easy to miss [15:38:23] I can't log into hosts that I deploy in tfinfratest after about an hour of them being online. I can log in at first, then I can't get in as myself or root. Any thoughts on what is happening? [15:39:27] can you use the console? [15:41:31] hmmm, this does not resolve to anything tf-infra-bastion.tfinfratest.eqiad1.wikimedia.cloud do I have a typo? [15:41:55] Looks like where I'm trying to go [15:42:43] I see the project name in horizon has caps in the name, maybe something got borked? [15:45:01] puppet cert exchange is failing on that host, I'm going to see if I can figure out why [15:45:35] but I am suspicious of the camelcase [15:45:47] I could make it without caps. Any suggestions on name? I was working to turn tf-infra-test into something without - characters [15:46:16] what about `tests` [15:46:43] or `cloudtests` [15:46:58] I may not be the best person to decide on naming :-P [15:47:16] I'm pretty sure that underscores are allowed [15:47:30] oops, maybe not [15:47:33] I think I have complained about every naming scheme we have, then proposed a new one, then complained about my own proposal years later [15:47:41] heh [15:47:49] Rook: mostly the thing to keep in mind is that the project name will be made into dns domains and urls [15:48:18] I figured that it would ignore the caps in those situations, but perhaps not [15:48:37] Yeah, maybe leave it as is for now so I can see where the hangup is [15:49:10] cloudvpstests? (as opposed to toolforgetests or similar) [15:49:20] the 'tf' was for terraform I believe [15:49:26] so it would be ot now [15:49:50] yep, I though we can use it for other tests too (and other with tools maybe at some point) [15:50:08] but maybe not, up to us [15:50:40] we already have 'cloudinfra' so maybe 'cloudinfratests' [15:52:31] sounds ok to me [15:52:48] tf is now for tofu :) [15:53:00] tofutests? [15:55:55] ok, I fixed puppet but dns is still messed up. I think best to empty and delete the project and start over without camelcase. [15:56:05] And also document/fix the cookbook which I will make a task for [15:56:36] (or switch to uuids) [15:56:52] * arturo offline [15:58:16] I'll deploy a new project name. It's fine to delete the camel case one? [15:58:29] Yes, but delete projects/volumes/etc before deleting the project [15:58:35] or there will be orphans which are hard to track down [15:58:41] Oh, good to know [17:15:17] * dcaro off