[08:01:40] <taavi>	 morning
[08:02:20] <dcaro>	 morning
[08:02:29] <arturo>	 o/
[08:03:23] <blancadesal>	 morning
[08:37:59] <dhinus>	 morning!
[08:46:11] <arturo>	 can I get a +1 to deploy this to toolforge and merge to main? https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/238
[08:51:48] <dcaro>	 do you want anyone to test it first?
[08:53:01] <taavi>	 it would be cool if that repo had a ci job that showed the diff of what would be applied to the clusters
[08:58:18] <dcaro>	 that'd be nice yes, it would need access of some sort to a k8s cluster no? Or can helm work the diff from a local file of sorts?
[09:00:36] <taavi>	 you can render the templates without access to the cluster. so render the templates for the main branch, render the templates for that commit, and show a diff for those, no cluster access required
[09:00:48] <taavi>	 of course that doesn't show if there's been any local hacks or so, but that is fine
[09:03:46] <dcaro>	 that'd be enough yes
[09:19:46] <arturo>	 there are a number of improvements we could add here
[09:20:13] <arturo>	 other that I would like to see is some kind of alert if there is a diff between this repo and what's live in the cluster
[09:20:41] <arturo>	 to help us detect undeployed changes
[09:20:53] <dcaro>	 there might be a task for that one
[09:21:16] <arturo>	 helm is not very good at detecting changes made by hand to the resources it manages, but I would like to have some detection for that as well
[09:21:31] <dcaro>	 T358908
[09:21:34] <stashbot>	 T358908: [infra,ci] Alert when toolforge-deploy changes are not deployed - https://phabricator.wikimedia.org/T358908
[09:21:42] <arturo>	 nice
[09:22:08] <dcaro>	 yep, helm ignores most (if not all) the changes made by hand, and keeps a secret (iirc) with the changes it deployed instead
[09:22:19] <dcaro>	 so if you change anything by hand, it does not notice
[09:22:57] <dcaro>	 and I think it was base64'd twice, for some reason
[09:23:27] <arturo>	 in the same line, having a cookbook, or script, to help force-redeploy may be useful
[09:23:41] <arturo>	 if a change is made by hand, running the deployment cookbook wont do nothing
[09:23:49] <arturo>	 so some kind of forced-redeploy is required
[09:36:54] <dcaro>	 uninstall-install of the chart works, but it deletes everything in the meantime (and we don't want that on toolforge)
[09:37:13] <dcaro>	 though probably we don't want to manually change anything on toolforge either
[09:44:13] <arturo>	 agreed
[09:58:05] <taavi>	 the CloudVPSDesignateLeaks alert has been flapping quite a bit over the weekend, does anyone know why?
[10:01:00] <dcaro>	 nope, though I saw a ticket fly by about some cloudvps project refreshing their VMs, so maybe they remove old VMs?
[10:02:03] <taavi>	 yeah but that should not flap the alert
[10:14:03] <dcaro>	 I though that old VMs were the only ones that would leak
[10:14:26] <dcaro>	 if they should not, then I don't know how the leaks would happen (besides bugs somewhere)
[10:19:47] <arturo>	 wouldn't it be cool for the k8s deploy cookbook to write a message to the merge request?
[10:23:23] <taavi>	 arturo: are the SAL messages it logs not enough? I'm worried that would result in too much spam
[10:24:30] <dcaro>	  I would find it useful to write it in all the bugs related to the release upgrade (so you know your fix has been deployed)
[10:24:38] <arturo>	 I usually write the message by hand into the MR anyway. I don't think is too much spam
[10:25:02] <dcaro>	 you still need to test that it works by hand though
[10:25:34] <dcaro>	 (so there might be a hand-written comment needed even if the deploy one is automated)
[10:26:49] <arturo>	 toolforge-deploy MRs can be updated and deployed multiple times (for example, when testing it), that's why I think it could be useful to have a record of when was it deployed (the updates are already tracked by gitlab itself)
[10:28:04] <arturo>	 without that message, using this as expample https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/238 you would need to stare at both the MR update history and the SAL and manually do the math to see when was each update deployed
[10:29:08] <arturo>	 I don't think this is high priority in anyway, just some information correlation I've missed a few times
[10:29:50] <dcaro>	 it might be more useful in that regard to have a 'live' way of seeing all the versions that are currently deployed in an environment
[10:30:05] <dcaro>	 (something I missed also several times already xd)
[10:30:47] <dcaro>	 mainly because deploys don't tell you the current state of the environment, as they can be reverted
[10:31:00] <arturo>	 something like this?
[10:31:02] <arturo>	 https://usercontent.irccloud-cdn.com/file/0aiEMHhO/image.png
[10:31:26] <dcaro>	 a bit nicer than helm list
[10:34:18] <dcaro>	 like highlighting mr-deployments, linking to the toolforge-deploy commit that was deployed with, etc.
[10:35:02] <dcaro>	 and filtering out user-charts
[10:35:28] <dcaro>	 grouping per component (instead of per helm chart)
[10:40:53] <taavi>	 I am rebooting cloudgw1002 (currently inactive eqiad1 cloudgw) for https://phabricator.wikimedia.org/T366555
[10:42:45] <arturo>	 ack
[10:43:03] <arturo>	 dcaro: in lima-kilo, when running the functional tests I get this
[10:43:17] <arturo>	 https://www.irccloud.com/pastebin/GAk5dne8/
[10:43:39] <arturo>	 triggered by the setup_file step of the build-smoke-test file
[10:43:41] <dcaro>	 I got that once, but went away the next time I tried to run it
[10:44:05] <arturo>	 I'm getting it always :-(
[10:44:06] <dcaro>	 I think it might happen when trying to cleanup if there's still no harbor project created for that tool
[10:44:24] <arturo>	 most likely is a deadlock
[10:44:37] <dcaro>	 we can add 'toolforge build clean --yes-i-know 2>/dev/null || :' to the tests to ignore failures
[10:44:38] <arturo>	 can't create the harbor project bc it can't run the tests
[10:44:49] <arturo>	 ok, will send that patch
[10:45:17] <dcaro>	 but probably the client should just not fail and say that no builds have been created yet
[10:45:38] <dcaro>	 though the FORBIDDEN returned by harbor is not helpful
[10:45:47] <dcaro>	 (instead of not found or such)
[10:46:51] <dcaro>	 it might be a change of behavior in harbor, as the api already did check that
[10:46:54] <dcaro>	 https://www.irccloud.com/pastebin/gqgfREu7/
[10:47:03] <arturo>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/321
[10:50:59] <arturo>	 thanks
[10:51:44] <taavi>	 arturo: to gracefully fail over from cloudgw1001 to cloudgw1002, I should stop keepalived on 1001, right?
[10:51:58] <arturo>	 taavi: mmmm
[10:52:08] <arturo>	 yeah, I think that should do it
[10:52:18] <arturo>	 they don't run BPG yet, right?
[10:52:21] <arturo>	 BGP*
[10:52:29] <dcaro>	 arturo: when rebooting the lima-kilo vm jobs-api starts failing with
[10:52:30] <dcaro>	 │ requests.exceptions.ConnectionError: HTTPSConnectionPool(host='kubernetes.default.svc', port=443): Max retries exceeded with url: /api/v1/namespaces/tf-public/configmaps/image-config (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x │
[10:52:36] <dcaro>	 (the python api)
[10:52:51] <taavi>	 arturo: correct
[10:52:54] <dcaro>	 [Errno -3] Temporary failure in name resolution
[10:52:54] <dcaro>	 xd
[10:52:56] <taavi>	 ok trying
[10:53:06] <arturo>	 taavi: anyway, a normal reboot should also be gracefully enough
[10:53:42] <arturo>	 dcaro: fails to resolve the k8s API FQDN itself, that's werid
[10:53:43] <taavi>	 yeah, I just wanted to be extra careful juuust in case there was something in the new kernel that required an immediate rollback
[10:53:46] <taavi>	 but seems like there is not
[10:53:54] <arturo>	 taavi: ack
[10:53:56] <taavi>	 now rebooting 1001
[10:54:06] <dcaro>	 arturo: it happens with some of the service actually, cert-manager, kyverno and metrics too fail to start the first time
[10:54:26] <arturo>	 dcaro: maybe they start before coredns itself is started
[10:56:13] <dcaro>	 kyverno is not coming up yet
[11:00:07] <arturo>	 downloading images?
[11:01:42] <dcaro>	 E0610 11:00:52.592068       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://10.96.0.1:443/api/v1/namespaces/kyverno/configmaps?fieldSelector=metadata.name%3Dkyverno-metrics&limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout     
[11:01:55] <dcaro>	 from the kyverno-admission-controller
[11:02:02] <dcaro>	 tohe 
[11:02:07] <dcaro>	 *note that I have 4 workers
[11:02:16] <arturo>	 that's should be the k8s API address
[11:03:08] <dcaro>	 let me re-delete the deployments for it again to forge recreating the pods
[11:03:19] <dcaro>	 *re-delete the pods to recreate the containers
[11:03:36] <dcaro>	 still the same error
[11:04:37] <arturo>	 is the k8s API up? did you load the base PSP?
[11:04:52] <arturo>	 did you run the ansible setup script in full?
[11:05:28] <dcaro>	 the ip might have changed after the reboot
[11:05:29] <dcaro>	 10.244.0.1/32
[11:05:41] <arturo>	 the k8s API address?
[11:06:04] <dcaro>	 the one I see from 'ip a' on the toolforge-control-node container
[11:06:24] <arturo>	 mmmm
[11:06:45] <arturo>	 so maybe that's self inflicted by the fact that we change the hostname and such after the VM creation?
[11:07:35] <dcaro>	 I think it might be a combination of multiple nodes + docker not keeping the networks after reboot
[11:07:43] <dcaro>	 let me try to reboot again, see if the ips change
[11:08:28] <arturo>	 but the k8s API address should remain stable, I expect it to don't be affected by stuff happening outside k8s
[11:08:30] <dcaro>	 hmm... though those ips are not allocated by docker
[11:09:32] <dcaro>	 hmm, maybe it's the docker inside the nodes (the one k8s uses)
[11:09:45] <arturo>	 that should be containerd
[11:10:04] <dcaro>	 they use containerd yep
[11:10:31] <dcaro>	 got confused by '--provider-id=kind://docker/toolforge/toolforge-control-plane' in the arguments
[11:20:24] <arturo>	 another weirdness dcaro 
[11:20:29] <arturo>	 in my system
[11:20:38] <arturo>	 @test "tail logs and wait (slow)" {
[11:20:38] <arturo>	     # this also waits for it to finish
[11:20:38] <arturo>	     toolforge build logs -f
[11:20:38] <arturo>	 }
[11:20:56] <arturo>	 this returns but the Status: OK is still not reported, so the next test fails
[11:21:04] <arturo>	 https://www.irccloud.com/pastebin/lA8kGgTp/
[11:22:19] <dcaro>	 you had a patch for that, I commented there, the issue is that the pipelinerun object might take a bit to get updated by the tekton controller, so even if the pod finished successfully, the object is not updated right away (eventual consistency means temporal inconsistency xd)
[11:23:05] <dcaro>	 about the extra k8s nodes, cordoning all but one worker makes everyone happy, so for some reason the network seems to get borked after reboot inside kind
[11:24:23] <dcaro>	 oh, but jobs-api fails with
[11:24:25] <dcaro>	 https://www.irccloud.com/pastebin/gDcwP2h9/
[11:24:37] <dcaro>	 is it trying to mount it from the worker node?
[11:25:06] <dcaro>	 it's there :/
[11:25:08] <dcaro>	 https://www.irccloud.com/pastebin/406agjF2/
[11:28:09] <dcaro>	 I'll rebuild the vm
[11:30:03] * dcaro lunch
[11:30:42] <dcaro>	 arturo: this is the mr for the build show issue: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/320
[11:48:36] <arturo>	 ok! I forgot
[12:47:36] <dcaro>	 oh, I just got the cpu issue!
[12:47:39] <dcaro>	 https://www.irccloud.com/pastebin/JkslKeno/
[12:47:47] <dcaro>	 clean VM, just ran the functional tests
[12:47:54] <arturo>	 🎉
[12:48:17] <arturo>	 maybe is not a CPU issue, but why is the node not available?
[12:48:34] <arturo>	 it might be related to the weird network condition after a reboot
[12:49:08] <dcaro>	 kyverno is taking ~20% of the cpu for itself
[12:49:37] <arturo>	 how did you see that?
[12:49:37] <dcaro>	 volume admission gets 14%
[12:49:43] <dcaro>	 kubectl describe node
[12:50:07] <dcaro>	 it has 10 pods, requesting 2% each (100m)
[12:50:43] <dcaro>	 jobs-api does not request anything xd (0%)
[12:50:48] <dcaro>	 maybe that's the solution?
[12:50:51] <arturo>	 mmm
[12:51:03] <arturo>	 this is request/limits not actual usage, no?
[12:51:19] <arturo>	 ok
[12:51:26] <dcaro>	 yep, that's what's getting exhausted
[12:51:29] <arturo>	 so maybe we can instruct kyverno to don't request anything in lima-kilo
[12:52:33] <dcaro>	 the others probably too
[14:59:20] <taavi>	 cloudweb puppet alerts are me, fix incoming
[14:59:38] <dcaro>	 ack, I was going to ask
[15:00:09] <dcaro>	 (git logs did not show anything very clearly xd)
[15:02:50] <taavi>	 fixed
[15:23:04] <bd808>	 As a result of the chat here last week about options for the redis container I had proposed I made <https://wikitech.wikimedia.org/wiki/Tool:Containers#Redis_container>. Feedback and testing welcome.
[15:27:17] <arturo>	 bd808: that's really cool!
[15:31:19] <bd808>	 the coolest part is that I just had to put together bits that y'all have made :)
[15:33:52] <andrewbogott>	 thanks bd808 !
[16:04:06] * arturo offline
[17:31:29] * dcaro off
[17:44:03] <bd808>	 This "containers" tool also gave me a good excuse to use Striker's support for multiple toolinfo records per tool that was noticed in this channel a few weeks ago-- https://toolsadmin.wikimedia.org/tools/id/containers