[07:19:46] <dcaro>	 okok cloudcephosd1006 is up and running
[08:31:38] <arturo>	 dcaro: found a little bug in the phabricator plugin for terminator https://www.irccloud.com/pastebin/Q8L2jryh/ it is missing a `T` character in the destination URL
[08:31:53] <dcaro>	 oh yes, solved it right after xd
[08:33:08] <arturo>	 👍
[08:34:42] <dcaro>	 I'm trying now to get it to highlight the link without mouse interaction, so with keyboard only
[08:34:51] <dcaro>	 but that's trickier xd
[08:36:26] <arturo>	 maybe you can at least change the color of the string? I'm not even sure if that's possible
[08:36:38] <arturo>	 like make it bold, red, underscore, or something similar
[08:37:44] <dcaro>	 yep you can, I'm having index issues (the link that's highlighted is different from the one that gets open)
[08:40:12] <dcaro>	 I used one of the existing plugins for starters that already does something similar
[08:53:43] <arturo>	 dcaro: could you please stamp this one? https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/49
[08:54:41] <dcaro>	 done
[08:54:42] <arturo>	 thanks!
[08:55:11] <arturo>	 heads up, deploying change to delete the per-tool-account PSP settings
[08:56:03] <arturo>	 in case you are wondering, there are at the moment 0 kyverno policy violations in the cluster
[09:04:06] <arturo>	 I deleted PSP from toolsbeta
[09:04:12] <arturo>	 but I cannot create jobs
[09:04:13] <arturo>	   Warning  FailedCreate  13s (x4 over 63s)  job-controller  Error creating: pods "once-with-retry-" is forbidden: PodSecurityPolicy: unable to admit pod: []
[09:04:25] <arturo>	 this is something I could not detect on my tests in lima-kilo
[09:04:39] <taavi>	 if the PSP controller is enabled, there must be a PSP to permit the pod creation
[09:04:49] <taavi>	 so the PSP controller needs to be disabled before deleting the PSP objects
[09:05:31] <arturo>	 I see
[09:05:47] <arturo>	 apparently I had the controller disabled in lima-kilo
[09:18:07] <arturo>	 taavi: I can manually edit the static pod manifests to un-load the PSP controller, but I have doubts about the kubeadm configmap. I want to make sure the next update don't add the controller again
[09:18:53] <arturo>	 I guess the question is: shall I just edit by hand the cm?
[09:29:20] <arturo>	 I'll do it
[09:33:58] <taavi>	 sorry, yes
[09:35:14] <arturo>	 everything looks good now, I'm running the functional tests in both toolsbeta & lima-kilo before proceeding with tools
[09:46:35] <arturo>	 heads up, disabling PodSecurityPolicy admission on tools
[09:46:52] <taavi>	 scary
[09:46:58] <arturo>	 :-S
[09:49:10] <arturo>	 done
[09:49:18] <arturo>	 now deleting per-tool-account PSP entries
[09:52:38] <arturo>	 jobs-api timed out for me in tools
[09:52:58] <arturo>	 now back online
[09:53:26] <arturo>	 taavi: could you please test random stuff in tools?
[10:31:01] <taavi>	 arturo: are you looking at 'Error: /Stage[main]/Kubeadm::Init_yaml/File[/etc/kubernetes/psp/base-pod-security-policies.yaml]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/kubeadm/psp/base-pod-security-policies.yaml'?
[10:52:57] <arturo>	 no
[10:53:09] <arturo>	 will fix now
[12:40:30] <andrewbogott>	 We warned: today I'm going to upgrade the cloudvirtlocal hosts to ovs, which means juggling etcd nodes. May result in alerts about degraded states &c.
[12:40:51] <arturo>	 ok
[12:41:48] <andrewbogott>	 *Be warned
[12:45:35] <dcaro>	 ack
[12:58:31] * arturo food
[14:00:40] <andrewbogott>	 taavi: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/10 please :)
[14:00:50] <andrewbogott>	 also https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1050368
[14:19:42] <dcaro>	 topranks: cloudcephosd1007 is having the same dhcp issues as 1006 had :/
[14:19:55] <dcaro>	 (when reimaging)
[14:24:16] <dcaro>	 yep, same, I manually ran link up and udhcpd and it worked, then continued with the install, and the network stopped working again
[14:25:27] <topranks>	 ok…
[14:25:56] <topranks>	 I’d love to say I’d had a shower thought of what might be going on since we last spoke.. but no
[14:26:07] <dcaro>	 I'm trying to rerun udhcpc, but it's failing now :/
[14:26:31] <topranks>	 link still showing up?
[14:27:08] <dcaro>	 yep
[14:27:26] <dcaro>	 on the host at least
[14:27:49] <dcaro>	 it says no carrier though 
[14:27:49] <dcaro>	 NO-CARRIER,BROADCAST,MULTICAST,UP
[14:28:30] <topranks>	 yeah that means it's down, but set to be up
[14:28:33] <dcaro>	 bring it down and up again it came up
[14:28:42] <dcaro>	 2: enp175s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
[14:29:04] <dcaro>	 and now it worked the dhcp xd
[14:29:49] <dcaro>	 and down again :/
[14:30:08] <dcaro>	 speed config issues?
[14:30:30] <topranks>	 switch side shows up 
[14:30:48] <topranks>	 nah you won't get those with 10G modules, they can only do 10/full
[14:30:59] <dcaro>	 it's up now again xd, manually flipped it off and on
[14:31:07] <topranks>	 only the older switches with RJ45 on the front you get that, the modular ones will stick to the speed the module inserted does 
[14:31:09] <topranks>	 or not work at all
[14:31:31] <dcaro>	 https://usercontent.irccloud-cdn.com/file/sCpDHRji/image.png
[14:31:34] <dcaro>	 some pings are very slow
[14:31:40] <topranks>	 switch side does show it bouncing up and down a few times 
[14:32:44] <dcaro>	 oh, it has no default gateway now :/
[14:33:15] <dcaro>	 I had to run dhcp again after flipping it off and on
[14:33:21] <dcaro>	 now it seems to work :l
[14:33:38] <topranks>	 this is no way to live
[14:33:45] <dcaro>	 I think that something in the way that debian installer configures the network (even if it's already done) messes it up?
[14:34:02] <dcaro>	 if I manually do that step and skip it seems to work
[14:34:13] <dcaro>	 but yep, that's not nice (got ~40 hosts to go...)
[14:34:28] <dcaro>	 and have to do it again relatively soon xd
[14:34:45] <dcaro>	 maybe the installer for bookworm does not have those issues?
[14:36:54] <dcaro>	 I'll stop and try to debug the next one if it happens again
[14:38:34] <arturo>	 taavi: did you manage to find the phab ticket about caching helm charts and container images?
[14:42:45] <topranks>	 dcaro: didn’t realise you were doing them all
[14:42:54] <topranks>	 yeah we’ll need to work it out
[14:46:34] <dcaro>	 I do them little by little, but they all need upgrading (first to bullseye, then bookworm+ceph version upgrade)
[14:47:33] <topranks>	 yeah.  ultimately I guess it's between dc-ops and i/f to work it out
[14:47:45] <topranks>	 I'll mention it Monday in our I/F meeting to make people aware and see if anyone has any ideas 
[14:49:47] <dcaro>	 ack, thanks, we'll see if it happens with the next, but it's two in a row so far
[14:53:28] <topranks>	 yeah if it was a one off we could maybe forget about it but looks like some new type of bug
[14:53:44] <topranks>	 next one maybe let’s get the firmware on the 100% known good one first
[14:59:35] <taavi>	 andrewbogott: sorry, was in meetings
[14:59:53] <andrewbogott>	 no worries! I self-merged, it seems to be working
[15:00:05] <taavi>	 i guess the resize api will work fine for local flavors, it won't try to migrate things or anything
[15:00:13] <taavi>	 migrate them between nodes that is
[15:00:34] <andrewbogott>	 actually it does! they must've finally fixed cold-migration
[15:39:23] <bd808>	 I missed your last meeting taavi :(( Thanks again for all the work and ideas and just being a cool person to interact with.
[15:43:09] <taavi>	 bd808: <3
[15:45:49] <arturo>	 taavi: I just created T368630 if you find an old one, please merge
[15:45:50] <stashbot>	 T368630: toolforge: make sure we cache in our repos/registries all helm charts and container images used in k8s - https://phabricator.wikimedia.org/T368630
[16:59:54] * dhinus off
[17:01:02] <dcaro>	 I just updated https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Ongoing_Efforts/Toolforge_Upgrade_Workgroup/Upgrades_Overview#Upgrade_tasks_and_major_changes and assigned tasks to people, the order was pseudo-random (first in the list with last, then second, then second to last...), feel free to change, the next round I'll do a different order to mix pairs up
[17:09:02] <dcaro>	 andrewbogott: I'm leaving cloudcephosd1007 getting added to the cluster, it will take some time, I'll re-check in a bit but if you see anything weird feel free to telegram me (or page me)
[17:09:26] <andrewbogott>	 ok!
[21:24:57] <dcaro>	 still rebalancing... might take a bit