[07:08:22] <dcaro>	 morning
[07:32:28] <blancadesal>	 morning
[08:21:18] <arturo>	 good morning
[08:21:30] <arturo>	 dcaro: what do you think about this?
[08:21:30] <arturo>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/26
[08:21:43] <arturo>	 I don't think we use anything similar for other components?
[08:23:11] <dcaro>	 not that I know of, that would require to create an alert of sorts so we don't overlook it being down right?
[08:23:19] <arturo>	 yes
[08:24:19] <arturo>	 I think I will create a phab ticket
[08:24:20] <dcaro>	 will partial runs mess with configmap data?
[08:24:36] <arturo>	 what I saw yesterday is:
[08:24:48] <arturo>	 1) initial run of maintain-kubeusers, it created some resources for some accounts
[08:25:18] <arturo>	 2) when it got an account that needed new PSP, it crashed, because somehow in toolsbeta there are some missing permissions for the daemon to be able to create PSP resources
[08:25:40] <arturo>	 3) the pod was restarted because the crash, and looped again the same accounts
[08:25:51] <arturo>	 4) until it found the same account with the missing PSP, then crashed again
[08:26:36] <arturo>	 it did not do anything weird to the configmap data, but it feel out of control to me. And maybe we want a full stop instead, so we can investigate and fix
[08:29:16] <dcaro>	 what about using backoffLimit instead? so it will still retry a few times (in case of temp failure)
[08:29:53] <arturo>	 is that available for deployments? I thought that was for jobs only
[08:30:23] <dcaro>	 maybe, let me check
[08:36:05] <dcaro>	 I think you are right, bummer
[08:36:36] <dcaro>	 what we do in maintain-dbusers, is just skip that account and report an error
[08:36:44] <dcaro>	 so we don't completely block new users, just that one
[08:37:02] <dcaro>	 maybe we can do that instead (and add the alert for the failed accounts)
[08:37:03] <arturo>	 mmm
[08:38:44] <dcaro>	 feels better than getting new user creation blocked completely until the bug is fixed
[08:38:56] <arturo>	 from a certain point of view, I like the idea of the dameon crashing and forcing us to fix the bug ASAP
[08:39:18] <arturo>	 daemon*
[08:41:36] <arturo>	 but we can definitely surface errors via prometheus and then alert on them
[08:42:40] <arturo>	 let me give it a spin
[08:42:53] <arturo>	 I'll report back after some tests
[08:44:21] <dcaro>	 that's what we did with maintain-dbusers, but it was very common for new accounts to get stuck for hours when any error happened, so we started just reporting the errors instead
[08:49:41] <arturo>	 ok
[09:34:16] <moritzm>	 hi, I'd merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/890001 in ~ 10mins, unless it's currently a bad time, just let me know if that's the case
[09:37:44] <arturo>	 moritzm: go ahead!
[09:40:08] <arturo>	 dcaro: I saw your MR https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 , is this for T357977 ?
[09:40:09] <stashbot>	 T357977: [toolforge.infra] create fullstack tests - https://phabricator.wikimedia.org/T357977
[09:44:53] <dcaro>	 arturo: kinda yes, though started as me being tired of running the same thing many times and copying into a script
[09:46:55] <arturo>	 I'm interested in something we can run locally, like that MR, and something that can report via prometheus
[09:47:40] <dcaro>	 Agree, that's why I split the 'lima-kilo' script from the tests themselves
[09:47:49] <dcaro>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/133/diffs#8ec9a00bfd09b3190ac6b22251dbb1aa95a0579d
[09:48:00] <dcaro>	 the tests in toolforge-deploy should be runnable from within any tool in toolforge
[09:48:15] <dcaro>	 while the script above does it for lima-kilo (re-using the tests from toolforge deploy)
[09:49:53] <arturo>	 ok
[09:57:27] <moritzm>	 ack, now merged
[10:33:01] <dcaro>	 arturo: quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/86
[10:33:14] <arturo>	 👀
[10:34:01] <arturo>	 dcaro: +1'd
[10:34:12] <dcaro>	 thanks!
[10:45:38] <dcaro>	 we might have overlooked that when we changed it everywhere else
[12:05:08] <arturo>	 dcaro: could you please help me make sense of the toolforge-tb-psp podsecuritypolicy ? what is it used for?
[12:06:17] <arturo>	 toolforge-tfb-psp*
[12:06:29] <arturo>	 https://www.irccloud.com/pastebin/HstS7mwY/
[12:11:26] <dcaro>	 👀
[12:12:52] <arturo>	 mmm I discovered google gemini using our repos as sources for the code examples
[12:12:54] <arturo>	 https://usercontent.irccloud-cdn.com/file/y1Alk6NV/image.png
[12:13:22] <dcaro>	 xd
[12:13:29] <arturo>	 https://g.co/gemini/share/b37cf50c8e96
[12:17:29] <dcaro>	 it seems to be the one specifying what tool pods should be restricted to no?
[12:18:23] <arturo>	 dcaro: there is another PSP for that. This is one was kind of "buildpacks-related" which is what is confusing me
[12:19:28] <dcaro>	 it's mentioned here operations-puppet/modules/toolforge/files/k8s/toolforge-tool-roles.yaml
[12:20:55] <dcaro>	 I think it's not used anywhere anymore
[12:21:02] <dcaro>	 it's from 2020 (before I was around)
[12:21:17] <arturo>	 ok
[12:21:27] <arturo>	 I will drop it as part of the refactor, then
[12:22:19] <dcaro>	 ack, it was added with https://phabricator.wikimedia.org/rLTMK844f21d480b65e1d58db3529e84576955ddccda6
[12:22:23] <dcaro>	 that we don't do anymore
[12:22:38] <dcaro>	 (we don't have a specific user for buildpack pods, it runs as the tool user)
[12:23:21] <arturo>	 ok
[12:25:00] <arturo>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1036640
[12:51:10] <arturo>	 do we have k9s for toolforge k8s?
[12:51:14] <arturo>	 I guess we need to package it
[12:51:25] <arturo>	 T366061
[12:51:26] <stashbot>	 T366061: toolforge: package k9s for use in kubernetes - https://phabricator.wikimedia.org/T366061
[13:01:36] * arturo food
[13:05:28] <dcaro>	 I think there's a copy of it somewhere in one of the control nodes that I used at some point
[14:27:21] <arturo>	 andrewbogott: are you working today? how do you disable a tool account in toolsbeta for testing purposes? I tried to modify the LDAP entry but I'm hitting stupids problems with LDIF and the tooling
[14:31:31] <andrewbogott>	 I am working and will see you in the checkin in a moment
[14:52:56] <andrewbogott>	 arturo, this might be an easy answer:
[14:53:01] <andrewbogott>	 andrew@cloudcontrol1003:~$ sudo mark_tool
[14:53:01] <andrewbogott>	 usage: mark_tool [-h] [--ldap-user LDAP_USER] [--ldap-password LDAP_PASSWORD]
[14:53:01] <andrewbogott>	                  [--ldap-base-dn LDAP_BASE_DN] [--project PROJECT] [--disable]
[14:53:01] <andrewbogott>	                  [--delete] [--enable]
[14:53:01] <andrewbogott>	                  tool
[14:53:18] <andrewbogott>	 hmmm it doesn't let you specify a different ldap server though does it
[14:53:31] <arturo>	 toolsbeta is the same LDAP server as tools
[14:53:59] <arturo>	 is just accounts are in the form of `toolsbeta.mytool` instead of `tools.mytool`
[14:55:09] <andrewbogott>	 hmmm ok
[14:55:21] <arturo>	 sudo mark_tool --project toolsbeta --disable test8
[14:55:26] <arturo>	 this did not report an error ^^^
[14:55:36] <andrewbogott>	 ok so it is easy, I'm just confused
[14:55:37] <arturo>	 but also, I don't see the LDAP entry being modified at all
[14:56:36] <andrewbogott>	 yeah, it's an extended flag which doesn't show up by default...
[14:56:43] <andrewbogott>	 what are you using to query?
[14:57:00] <arturo>	 aborrero@mwmaint1002:~ $ ldapsearch -x uid=toolsbeta.test8
[14:57:40] <arturo>	 maintain-kubeusers also doesn't detect the account to be disabled
[14:58:08] <andrewbogott>	 hm
[15:02:59] <andrewbogott>	 I'm still digging
[15:04:20] <andrewbogott>	 arturo: I see two easy ways to check if mark_tool  --disable did its business.
[15:04:28] <arturo>	 ok
[15:04:56] <andrewbogott>	 here's one:
[15:04:59] <andrewbogott>	 andrew@mwmaint1002:~$ ldapsearch -x uid=tools.andrewtesttoolfour  | grep loginShell
[15:05:00] <andrewbogott>	 loginShell: /bin/disabledtoolshell
[15:05:02] <andrewbogott>	 here's the other:
[15:05:28] <andrewbogott>	 andrew@mwmaint1002:~$ ldapsearch -x uid=tools.andrewtesttoolfour + | grep disabled 
[15:05:28] <andrewbogott>	 pwdPolicySubentry: cn=disabled,ou=ppolicies,dc=wikimedia,dc=org
[15:05:31] <arturo>	 oh! I see loginShell: /bin/disabledtoolshell
[15:05:47] <andrewbogott>	 That '+' shows the hidden things
[15:06:09] <arturo>	 great, I see stuff now
[15:06:11] <andrewbogott>	 there's also
[15:06:13] <arturo>	 thanks!
[15:06:14] <andrewbogott>	 andrew@mwmaint1002:~$ ldapsearch -x uid=tools.andrewtesttoolfour + | grep pwdAccountLockedTime
[15:06:14] <andrewbogott>	 pwdAccountLockedTime: 20240528150217.539103Z
[15:06:17] <andrewbogott>	 yep
[15:06:32] <andrewbogott>	 That dangling + is such weird syntax that I have to re-learn it every time
[15:06:46] <arturo>	 ok, so I think the news here is that maintain-kubeusers is not correctly detecting disabled accounts
[15:09:42] <andrewbogott>	 That's possible although I'm sure that the subsequent steps were happening properly last time I checked
[15:09:45] <andrewbogott>	 (which was a while ago)
[15:10:04] <andrewbogott>	 arturo: I think it also checks a lockfile before doing anything
[15:10:21] <arturo>	 yeah
[15:10:26] <arturo>	 all that was refactored
[15:10:46] <arturo>	 but this function
[15:10:48] <arturo>	 https://www.irccloud.com/pastebin/dpeohy1f/
[15:10:56] <arturo>	 is what I suspect is not working as expected
[15:11:23] <andrewbogott>	 hm, that should be the easy part
[15:11:24] <arturo>	 I'll double check all that logic soon
[15:11:33] <arturo>	 thanks for the assistance!
[15:12:21] <andrewbogott>	 np
[15:32:10] <blancadesal>	 rook: how much effort do you think it would take to integrate this with PAWS, if at all feasible?: https://quarto.org/ (asking on behalf of a volunteer)
[15:32:30] <Rook>	 Can answer in an hour, in a meeting
[15:32:48] <blancadesal>	 thanks, no rush!
[16:11:53] * arturo offline
[16:21:36] <andrewbogott>	 Rook: do you have time/patience to walk me through finishing the paws deploy that you just started? Or want to ping me after you eat lunch?
[16:23:06] <Rook>	 Sure we can walk through it
[16:23:59] <andrewbogott>	 ok. this will be very tedious as I don't spend much time in github.  it looks to me like the current issue is fixing the pip issue in https://github.com/toolforge/paws/actions/runs/9272318205/job/25509784853 
[16:24:55] <Rook>	 Yeah that looks like the problem. I suspect it is because it went past python 3.11 and now wants a virtual env or sys packages
[16:25:00] <Rook>	 Want to go through it on call?
[16:25:48] <andrewbogott>	 yep, sure
[16:26:15] <dcaro>	 oh, that's pip refusing to install stuff system-wide, I guess you are not using a venv?
[16:36:59] * dcaro off
[16:37:00] <dcaro>	 cya
[19:17:05] <Rook>	 d.caro for tomorrow or other good work times if you see this. Do you have an opinion on using the unappealing looking --break-system-packages for containers running pip? I feel like it doesn't matter, as I'm fine with containers having pip installed packages over system packages. But is there something I'm missing?