[10:56:17] <jbond>	 hi all anyone able to look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/845027 (removing the id key from the wikimedia_clusters data structure in hiera) i think its harmless but possible there is something lurkine somehwere that uses this
[12:27:11] <claime>	 I had a bad state on kubernetes2006.codfw.wmnet that meant I had to reboot it through the single-node reboot cookbook while the k8s reboot node cookbook was running. It came back correctly, I uncordoned it and repooled it through confctl, anything else I should do? I don't think so but I'd rather check
[12:30:16] <jayme>	 claime: no, that should be it. What was "bad state", though?
[12:31:37] <claime>	 jayme: network interface systemd service was in a failed state 
[12:31:39] <claime>	 Oct 21 11:41:28 kubernetes2006 systemd[1]: Started ifup for ens13.
[12:31:41] <claime>	 Oct 21 11:41:28 kubernetes2006 sh[14683]: RTNETLINK answers: File exists
[12:31:43] <claime>	 Oct 21 11:41:28 kubernetes2006 sh[14672]: ifup: failed to bring up ens13
[12:31:45] <claime>	 Oct 21 11:41:28 kubernetes2006 systemd[1]: ifup@ens13.service: Main process exited, code=exited, status=1/FAILURE
[12:31:47] <claime>	 There was nothing actually wrong I could see
[12:32:15] <claime>	 Just I couldn't rule out something not being configured correctly, so I opted for a reboot
[12:33:05] <claime>	 (by nothing wrong, I meant the adresses were up, the routes were good, but ifstate returned 1)
[12:33:05] <jayme>	 ah, that's probably https://phabricator.wikimedia.org/T273026
[12:33:19] <jayme>	 verry annoying
[12:33:58] <claime>	 Oh there's actually a non-rebook solution lol
[12:34:02] <claime>	 reboot*
[12:35:21] <claime>	 Added a SAL log pointing to it
[12:35:32] <claime>	 Just so we know :)
[12:40:52] <jayme>	 👌
[12:41:28] <claime>	 I'm gonna reboot kubernetes2023.codfw.wmnet and kubernetes2024.codfw.wmnet that aren't in the list because they're insetup
[12:41:44] <claime>	 Might as well :p
[12:44:16] <jayme>	 yeah, I thing they are assigned to infra foundations because of insetup :-p
[12:48:05] <claime>	 lol
[13:29:44] <inflatador>	 <o/
[15:56:37] <claime>	 Most kube nodes rebooted, a few need to be dealt with manually but are operational. Our list of hosts on the task is up to date.
[15:56:43] <claime>	 Have a good week end