[01:13:19] !log k8splay - deleted 2 gitlab related instances (not the ones in the actual gitlab project) [01:13:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:K8splay/SAL [08:35:07] Sorry here I am. majavah: I ssh into my toolforge profile @tools-sgebastion-07, I then try sudo apt-get update since the python version needs updating and upgrading. The system asks me for the pw for . Honestly it's not the first time that I run a sudo command and I am asked to re-imput my password (not in the toolforge env, I mean in general in the past) [08:42:36] CrissG: you can't run apt in a tool [08:42:45] It would effect everyone [08:42:59] What Python version do you need and why [10:52:13] !log project-proxy create new pair of proxies on bullseye T295235 [10:52:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL [10:52:16] T295235: Update project-proxy proxies to Debian Bullseye - https://phabricator.wikimedia.org/T295235 [14:21:32] RhinosF1: Oh I see. Apologies for that then, I thought it would have affected my repo/partition only. The current python version available is 3,5, though there was a compatibility issue wrt string embedding using the f option (i.e. f'string content {value}'). Anyway in the meantime I managed to use alternative embeddings and everything is working fine. I thought of updating the python [14:21:32] version because I thought it was just for my partition. [14:22:02] Thank you for your help! [15:21:48] CrissG: newer versions of python3 are available as Kubernetes containers in Toolforge. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Available_container_types The bastions and grid engine will stay at python 3.5 until we upgrade the operating system there from Debian Stretch to something newer (Buster or Bullseye). [15:31:40] What bd808 said, a venv will work [15:31:51] * RhinosF1 was in college and is now prepping for an interview [16:24:44] !log paws deployed PR97 (85c085f5309d45c598c3089fd64fa4810c769c06) Update Pywikibot to 6.6.2 [16:24:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:46:15] !log machine-learning Created project with 32 cores, 128GB RAM T294964 [16:46:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Machine-learning/SAL [16:46:18] T294964: Request creation of machine-learning VPS project - https://phabricator.wikimedia.org/T294964 [16:46:31] Forwarded from unknown: 出售邮箱无限发送软件，日发5w封。 [16:46:32] 不封号，不屏蔽，百分百送达邮箱。 [16:46:33] 可发国内国外邮箱，可做灰色项目。 [16:46:35] 送邮箱采集器，精确采集。 [16:46:36] 支持任何测试 [16:46:38] 接邮箱，飞机代发，联系 @LP7744 [16:55:20] bd808 and RhinoF1: Makes sense, I'll look into that then, thanks. I'm still new to toolforge and the good practice for developing in it [16:55:29] Forwarded from unknown: 出售邮箱无限发送软件，日发5w封。 [16:55:30] 不封号，不屏蔽，百分百送达邮箱。 [16:55:32] 可发国内国外邮箱，可做灰色项目。 [16:55:33] 送邮箱采集器，精确采集。 [16:55:35] 支持任何测试 [16:55:36] 接邮箱，飞机代发，联系 @LP7744 [16:56:33] bd808: ^^^^ [16:56:34] * bd808 kickbans the telegram spammer [16:56:42] * majavah is not sure who else has admin rights on telegram [16:57:48] !log k8splay set --cores 36 --ram 72000 --instances 20 T295376 [16:57:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:K8splay/SAL [16:57:51] T295376: Request increased quota for k8splay Cloud VPS project - https://phabricator.wikimedia.org/T295376 [16:58:55] majavah: there are quite a few patrollers and admins on the telegram side, but we don't have a ping convention to alert them all. [20:51:36] Hello, I could use some help from admins with my newly created toolforge tool. Is this the right channel here? [20:55:01] Silvan_WMDE: yea, the place is good. go ahead and ask the actual question and wait a little bit :) [20:57:08] ok, I managed to mess up the tool's home directory and delete the .kube folder by accident. Now the webservice command refuses communication, complaining "No such file or directory: '/data/project/mismatch-finder-staging/.kube/config'" [20:57:53] is there any hope to fix this or should I look for ways to delete the tool and re-create? [21:06:05] !help could a toollabs admin restore a .kube dir for a tool? ^ [21:06:05] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [21:11:03] Silvan_WMDE: while we wait for that, have you tried recreating just the empty .kube dir yourself? and see if by any chance it recreates the config [21:13:20] I haven't tried that, but I think what's more tragic is the fact that this config holds references to the client-certificate and client-key files in .toolskube/, which I also accidentally deleted [21:13:40] I know I used to know how to fix this... [21:13:57] * bd808 looks on wikitech for magic commands to run for Silvan_WMDE [21:14:04] :) [21:14:26] loves how the !help trigger worked so quickly [21:14:50] I should go back and read tht rsync man page again on how to not accidentally delete entire directories [21:15:23] Silvan_WMDE: add an -n to the rsync command makes it safe and turns everything into a dry-run, then when you like it, drop the 'n' [21:17:25] I think `--exclude=".*"` would keep rsync from messing with the various dotfiles. [21:18:22] mutante: I should have done that, definitely [21:19:02] bd808: my plan was to rsync into a sub-dir. don't even know how I managed to end up one level higher [21:20:01] `kubectl delete cm maintain-kubeusers` is the magic incantation to tell maintain-kubeusers to rebuild credentials for a tool -- https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Certificates#operations_4 -- although the instructions there only work if you have working credentials already. :) I will do the needed stuff using magic admin super powers. [21:22:32] oh, interesting, I would not have guessed this would be a kubectl command [21:22:38] thanks bd808 [21:22:43] bd808: so you probably need the tool's name? [21:22:57] Silvan_WMDE: mismatch-finder-staging? [21:23:08] ah, yeah - it's mentioned above in the error msg [21:23:26] bd808: correct [21:23:45] mutante: that command is removing the bit that the custom Kubernetes service checks for to decide if the tool is setup already or not. [21:24:03] when it's gone the service will "see" the tool on its next run [21:26:32] !log tools.mismatch-finder-staging `kubectl delete cm maintain-kubeusers -n tool-mismatch-finder-staging --as-group=system:masters --as=admin` to trigger regenerating credentials [21:26:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mismatch-finder-staging/SAL [21:27:27] Silvan_WMDE: if you don't have new credentials in ~10 minutes or so ping again. :) [21:27:55] bd808: great, thank you very much! [21:29:37] bd808: *nod* ty! [21:30:06] polling etcd for status is faster than polling NFS files :) [21:32:41] these were quick 10 minutes... credentials are back and I can start/stop/restart the webservice again, thx! [21:33:24] now let's see how long it takes to re-do my work I did today... :-/ [21:33:34] there is a wikitech user where I saw with my own eyes what email address they use per wikitech.. but I can't find them in ldapsearch and that seems such an odd combo [21:33:53] also they can login on wikitech but not on horizon. hmmm [21:34:21] mutante: is there a phab task? [21:34:30] no, I have to make it [21:34:45] it came up today while I was doing some kind of workshop using cloud VPS [21:35:07] will do [21:35:23] I know you can have horizon but not wikitech [21:35:36] But I'd imagine the other way round is impossible [21:35:42] a user that exists on wikitech, but not in LDAP is not impossible but is unlikely [21:36:26] I think horizon will only let you in if you are an admin is at least one Cloud VPS project, but I could be wrong about that [21:36:42] oh, I know that part, I just tested that today with 10 people :) [21:36:55] mutante: can they get into gerrit or anything? [21:37:06] it will show you some error about not being able to pull quotas for bastion or so, but otherwise they are logged in [21:37:25] then I could add them to a project, tell them to logout and login again, and it's alright [21:38:00] *nod* so I guess you would just have to be a member of at least one Cloud VPS project so that you have a Keystone user (OpenStack auth service) [21:38:01] RhinosF1: haven't asked them about gerrit yet, but good idea [21:38:44] the special user is fgoodwin@ [21:38:46] If they can get into gerrit then ldap must exist [21:40:53] mutante: that's uid=frankie,ou=people,dc=wikimedia,dc=org. cn=Fgoodwin [21:41:19] I used `'mail=fgoodwin@*'` as my ldapsearch query [21:41:56] User:Fgoodwin [21:41:58] on wikitech [21:42:45] *nod* that's the cn=Fgoodwin part of my response. cn in LDAP == wikitech username [21:42:56] bd808: OOOH MAN.. you know what this is? again? [21:43:04] I can see them [21:43:08] I use "email" where I need to use "mail" [21:43:15] and get 0 results [21:43:20] And they are in wmf group [21:43:22] ha. I've done that too :) [21:43:26] and don't notice the difference between wrong field and "user does not exist" [21:43:30] because there is none in the result [21:44:31] ok, I see the user, one part solved, and thank you. but now ..why cant they login on horizon but the others could [21:45:33] they got an error that would seem to match "not in any project" but it wasn't the same for other users [21:45:50] but let me just add her user to the project either way [21:47:18] done and I will let her test again [21:51:09] I see `memberOf: cn=project-k8splay,ou=groups,dc=wikimedia,dc=org` in their LDAP record now, so it seems like it should work. [21:52:38] great! thanks for checking [22:07:19] bd808: RhinosF1: resolved. it works for her and she can login after I added her to a project, it did seem different from others but maybe I was wrong and they all had another project or the difference is that they had logged in before at some point. all good now. [22:13:51] np