[09:54:28] claime: following up on your email yesterday, reimaging those wikikube-worker nodes failed for you with the (now default) tftp? [10:22:26] topranks: it failed for all but one of them yeah [10:23:04] hmm ok. merits further investigation but seems we just reversed the curse :( [11:37:49] Note that I'm deploying new parsercache key spec gradually, that will trigger a lot of reparses (but not too large to bring down everything) so latency and many other metrics will jump but it shouldn't be concerning [12:08:06] <_joe_> so SNAFU is what you're saying? [12:08:12] <_joe_> did I understand that correctly? [13:06:05] hnowlan kamila_ , I think I have successfully added a silence for the kafka hosts I am migrating, I have to run for a bit and will be back, if an alert fires, I am so so so sorry guys [13:06:17] ack, thanks! [16:05:20] taavi: are you ok if I puppet merge your changes? [16:05:21] yes please [16:06:43] cheers [16:36:17] Amir1: is it trivial to upgrade the map on https://fault-tolerance.toolforge.org/map? And, any idea why cloudvirts and cloudcephosd nodes don't appear there? (I looked briefly at the netbox query but I'm missing something) [16:41:27] fabfur is it OK if I merge your magru DNS change? [16:42:07] inflatador: please do [16:42:12] he is on a break [16:42:27] sukhe ACK, merging now [16:56:28] andrewbogott: give me a second, I update it [17:01:08] it should be updated now [17:05:08] thank you! [17:05:31] still no cloudvirts [17:30:29] is it in any cluster? I think the problem for cloudvirts are that they are not listed as pybal clusters, so it can't get the list of hosts to build the map [17:31:05] https://config-master.wikimedia.org/pools.json [18:02:47] <_joe_> tbh that should be promoted to be a prod thing under IDP and running on aux k8s :) [18:59:13] Amir1: oh, I didn't realize that was pybal only, that explains it [19:17:50] _joe_: patches are welcome 😁 [19:18:36] andrewbogott: It’s not too hard to add other types of clusters as long as there is a way to load them. See the source code, it's quite basic [19:19:02] ok! [19:19:02] Just finding a way to list the host names is enough [19:24:59] more specifically, this: https://gitlab.wikimedia.org/toolforge-repos/fault-tolerance/-/blob/main/fault_tolerance/clusters.py?ref_type=heads