[08:22:25] greetings [08:23:59] morning [08:24:28] Greetings [08:24:58] dcaro: I want to rebuild the toolforge prebuilt images [08:25:51] It’s shouldn’t be hard, and it’s just to add the newly merged changes to images [08:26:41] But given that it touches a lot of things I wonder if there are any specific protocol or can just log into the imagebuilder hosts and just do that? [08:28:58] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Update_prebuilt_images there's a bit of docs there [08:29:35] Ok thanks [08:30:16] I would highly recommend testing a few of them, both with webservices and jobs [08:30:57] (functional tests probably only use one or two pre-built images, so they are ok, but imo don't give too much confidence) [08:31:48] Yeaaa, let me proceed with it [08:32:27] with image-config, you can test first in toolsbeta, then tools [08:35:48] Maybe build with a :test tag and clean that up later after testing [08:37:20] morning [08:39:55] Raymond_Ndibe: I did update one of those not long ago, and it worked fine, more details at https://phabricator.wikimedia.org/T423417#11930268 [08:43:25] dhinus: thanks! [09:21:38] fun tofu changes for someone to look at: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/354 throught 356 [09:52:37] dhinus: I draw this the other day about toolforge deployment/build/job concepts and such, might help clarify ideas (not perfect, not exhaustive, but might help) https://drive.google.com/file/d/1ttiZYgiW1-j1XOU3CBs1z4SR-Of2XHZR/view?usp=drive_link [09:52:43] taavi: very nice, LGTM [09:54:15] thanks! [10:05:48] sure np [10:27:41] dcaro: thanks, that's useful! [11:05:45] dhinus: let me know if you want me to take the config side of the alerts opt-in [11:07:18] dcaro: yes please, unless it delays more important/urgent things on your hypothesis [11:08:51] I would suggest a quick discussion in the phab task to define the desired structure in the config ("global.alerts" vs "alerts" vs something else) [11:10:07] Ack [11:10:48] Feel free to propose something if you have anything in mind, otherwise I'll write down a couple options see which one looks better [11:10:58] (after lunch) [11:33:05] rabbit_holes++ [11:34:51] I assume no-one relies on the puppet-managed clone of maintain-kubeusers in /srv/git on all tools k8s control nodes ever since that was moved to toolforge-deploy, shout at me if that is an incorrect assumption [11:35:05] (yes, that is related to setting up an etcd cluster in cloudinfra. don't ask me why) [12:16:50] (I will not :D ) [12:16:54] * volans errand to run, bbiab [13:02:13] taavi: I think nobody does, might have been for manual testing? [13:04:53] proxy-7/8.project-proxy alerts are me, new instances being set up, please ignore while I try to silence them all [13:49:54] ack [15:20:07] cloudinfra-db backups: https://gerrit.wikimedia.org/r/1342681 https://gerrit.wikimedia.org/r/1342694 [15:26:25] andrewbogott: fyi https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143/diffs (current triggering alert) [15:29:06] which alert? [15:30:12] oh, you just acknowledged it xd [15:31:14] so the gitlab link was a non sequitur? [17:27:35] oh, sorry, that was meant to be a grafana link, I pasted the wrong thing xd [17:33:48] * dhinus off [17:54:38] * dcaro off [18:04:22] what happened to the 'needs review' gitlab label?