[07:48:36] <dcaro>	 morning
[07:49:04] <taavi>	 o/
[07:50:34] <arturo>	 o/
[07:59:19] <blancadesal>	 goooood morning
[08:22:06] <dhinus>	 o/ happy friday :)
[08:42:47] <dcaro>	 quick review? (blocks ci for some projects) https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/33
[08:44:37] <dcaro>	 hm, I think I might able to use 'include: local: ...' and avoid one misdirection
[08:47:51] <arturo>	 dcaro: LGTM
[08:48:41] <dcaro>	 thanks! I can use local it seems :), I'll merge this one, and send a new one using 'local' for all the includes (that helps a lot testing things as otherwise you have to add `ref: branchname` to every include to test a branch)
[08:51:22] <blancadesal>	 https://www.irccloud.com/pastebin/LiT8uTW8/
[08:52:31] <taavi>	 hmm, why is that using tools-harbor.wmcloud.org for the dev chart and not toolsbeta-harbor.wmcloud.org?
[08:52:48] <blancadesal>	 that's my question :)
[08:53:04] <blancadesal>	 in the second case, it does use toolsbeta
[08:53:52] <dcaro>	 that's defined in the toolforge-deploy repo
[08:54:38] <dcaro>	 probably has the wrong repo source defined there
[08:55:01] <dcaro>	 (it's a non-often updated component, so we probably never noticed)
[08:55:27] <blancadesal>	 i'll send a fix
[08:55:32] <dcaro>	 👍
[08:59:38] <blancadesal>	 dcaro: should the chart repository in all components' local.yaml be toolsbeta? (i'm seeing a few others where it's tools)
[09:00:25] <dcaro>	 I think so yes
[09:00:33] <dcaro>	 hmm
[09:01:29] <dcaro>	 we should sit down and figure out the whole thing at some point (as in, do we want to deploy toolsbeta images on toolsbeta, or tools-harbor ones? as we are testing right before deploying to tools?)
[09:04:32] <arturo>	 why woould `toolforge-jobs logs contjob -f` stop following the streaming and return for no apparent reason? I can reproduce in lima-kilo
[09:04:54] <taavi>	 some timeout somewhere?
[09:05:12] <arturo>	 maybe, the job is only producing a log entry every 30 seconds
[09:06:27] <dcaro>	 I think that there's a task for it somewhere
[09:07:17] <dcaro>	 my guess is that there's a timeout somewhere yes
[09:08:09] <dcaro>	 I think https://phabricator.wikimedia.org/T359953 might be the same, but now we don't show the ugly errors
[09:08:21] <dcaro>	 (that was because of a version incompatibility iirc)
[09:08:24] <blancadesal>	 quick review: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/249
[09:09:11] <arturo>	 blancadesal: LGTM
[09:09:22] <arturo>	 dcaro: ACK
[09:09:30] <blancadesal>	 arturo: thanks
[09:17:49] <dcaro>	 btw. this is ready to review https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/34 (have to remove the last commit before merging, but keeping it until then otherwise the task fails)
[09:18:16] <dcaro>	 that's the pre-commit autoupdate pipeline thingie
[09:19:23] <arturo>	 dcaro: LGTM
[09:20:15] <dcaro>	 I think I pasted the wrong one xd, the one I was talking about is https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/32 (thanks for the other review too tohugh)
[09:21:54] <blancadesal>	 sneaky way to get your stuff reviewed :P
[09:22:05] <dcaro>	 sorry :S
[09:22:54] <blancadesal>	 was just kidding :))
[09:26:59] <blancadesal>	 dcaro: lgtm
[09:28:12] <dcaro>	 👍 next step is add the pre-commit caching to speed up the ci, hopefully chopping the time from hours (it timed out at 1h a few times) to less 1 min (my goal xd)
[09:34:42] <arturo>	 xD
[09:39:19] <dcaro>	 ex. https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/jobs/240111 -- ~15 min running so far
[09:42:00] <arturo>	 too many safety checks, maybe we should write less safe code :-P
[09:42:56] <dcaro>	 it would definitely have it's advantages yes :)
[09:50:32] <dcaro>	 oh, I think that the local includes don't work as I expected :/, https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/pipelines/48508
[09:50:47] <dcaro>	 even though the tests I made using the custom branch seemed to work
[09:50:53] <dcaro>	 I'll revert
[09:54:56] <dcaro>	 hmpf... it seems it might not be possible :/ https://gitlab.com/gitlab-org/gitlab/-/issues/35180
[09:55:13] <dcaro>	 anyhow, will look at it another time, focusing on the pre-commit issues
[09:55:43] <blancadesal>	 what was the way to curl the api-gateway from within lima-kilo? 
[09:56:02] <dcaro>	 you can check the /etc/toolforge.yaml 
[09:56:05] <dcaro>	 file for the endpoint
[09:56:52] <dcaro>	 then you need to pass the --cert /home/tf-test/.toolskube/....cert --key <similarpath>.key (don't remember the full path, might be private.cert instead of key or something like that)
[09:57:38] <dcaro>	  /data/project/tf-test/.toolskube/client.crt and client.key
[09:58:09] <blancadesal>	 ah yes it's /etc/toolforge/common.yaml
[09:58:10] <dcaro>	 and /etc/toolforge/common.yaml xd I'm the worst remembering details
[09:58:32] <dcaro>	 (never had to with autocomplete...)
[09:58:43] <blancadesal>	 we still have minikube instructions in all the readme's 
[09:59:02] <dcaro>	 oh, we might want to sort that out
[10:02:59] <arturo>	 please review and approve: https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/6
[10:03:27] <blancadesal>	 local.tf-test@lima-lima-kilo:~$ curl --cert /data/project/tf-test/.toolskube/client.crt --key /data/project/tf-test/.toolskube/client.key --insecure 'https://localhost:30003/oapi/'
[10:03:27] <blancadesal>	 {"message":"Hello World"}
[10:03:30] <blancadesal>	 yay
[10:03:33] <taavi>	 \o/
[10:04:59] <dcaro>	 \o/
[10:05:07] <arturo>	 🎉
[10:06:33] <blancadesal>	 I learned some plumbing skills, xd
[10:09:49] <dcaro>	 arturo: approved
[10:10:46] <arturo>	 dcaro: thanks
[10:10:52] <arturo>	 please review and approve please review and approve https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/41
[10:11:05] <dcaro>	 xd, pretty pretty please
[10:12:14] <dcaro>	 that one will take a bit more
[10:12:27] <arturo>	 heh, copy paste error
[10:15:49] <dcaro>	 running this on tools-k8s-control-9 gets mee too many open files
[10:15:50] <dcaro>	 root@tools-k8s-control-9:~# kubectl -n envvars-admission logs deployment/envvars-admission -f
[10:16:15] <dcaro>	 ulimit -a show 1024 open files limit only
[10:16:39] <dcaro>	 I think might be related to the build issues people were seeing lately
[10:18:43] <dcaro>	 it's running on workers 102 and 103
[10:20:05] <arturo>	 dcaro: I think that line is actually in the log file, no?
[10:20:17] <dcaro>	 running without -f does not show it
[10:20:41] <arturo>	 right
[10:21:08] <dcaro>	 it seems our file open limit on all the nodes is 1024, that seems low
[10:22:16] <arturo>	 this may be https://github.com/kubernetes/kubernetes/issues/64315
[10:23:38] <dcaro>	 https://www.irccloud.com/pastebin/JseCHxWg/
[10:23:43] <dcaro>	 it only gives 6 on each xd
[10:24:24] <dcaro>	 before and after the logs command on the control node
[10:25:41] <arturo>	 maybe the limit is from within the container itself
[10:25:43] <arturo>	 not the node?
[10:27:32] <dcaro>	 probably yes, on one of the workers, the number of open files per-pid
[10:27:43] <dcaro>	 https://www.irccloud.com/pastebin/4CG8k88j/
[10:27:55] <dcaro>	 (only the 14 on the top)
[10:28:04] <dcaro>	 there's processes using >1024 there
[10:28:56] <dcaro>	 that last one is containerd-shim
[10:31:57] <dcaro>	 https://www.irccloud.com/pastebin/Dyl68nEY/
[10:32:05] <dcaro>	 not even close to it
[10:35:58] <dcaro>	 interesting
[10:36:03] <dcaro>	 https://www.irccloud.com/pastebin/Xlib1kSA/
[10:36:11] <dcaro>	 I'll open a task/find the one already there
[10:38:27] <arturo>	 weird!
[10:44:58] <dcaro>	 it's the worker only, tailing the pod on the other worker works well
[10:47:09] <dcaro>	 the other worker does not have errors on journalctl either
[10:59:57] * dcaro lunch
[12:27:03] <dcaro>	 got a patch for the open files issue: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1019277
[12:27:21] <arturo>	 :eyes
[12:27:22] <arturo>	 :
[12:27:55] <arturo>	 dcaro: LGTM
[12:29:09] <dcaro>	 thanks :)
[12:29:35] <dcaro>	 ^ that wasn't me :/
[12:29:40] <dcaro>	 (did nothing yet)
[15:12:34] <bd808>	 thanks for digging into that fsnotify bug dcaro. I will keep my eye open for more problems but hope that the limit bump fixes it. :)
[15:51:09] * arturo offline
[17:40:04] * bd808 lunch
[23:16:50] * bd808 off