[08:02:05] <marostegui>	 Why is wikimedia-sre not listed at https://meta.wikimedia.org/wiki/IRC/Channels?
[08:06:45] <volans>	 noone thought about it? :D
[08:07:52] <zabe>	 yeah, "This page attempts to track and identify the numerous Wikimedia IRC channels"
[08:07:56] <zabe>	 attempts
[08:09:22] <marostegui>	 I'll add it
[08:10:36] <volans>	 +1
[08:11:32] <Emperor>	 here was me thinking it was our secret club :)
[08:36:08] * vgutierrez hides "the good stuff"
[08:48:19] <Emperor>	 that explains why I have to look after swift ;p
[11:48:44] <effie>	 Dear SREs who do k8s, I will attempt to deploy a dangerous change in admin, please refrain from doing anything dangerous until we are done, thank you !
[12:14:51] <fabfur>	 oof, I wanted to go to the zoo feeding tigers bare handed. Well, I'll postpone it...
[12:34:22] <kormat>	 :D
[12:37:33] <elukey>	 Emperor: o/
[12:37:58] <elukey>	 would it be ok for you if I upgrade the swift eqiad's envoys to PKI>
[12:37:59] <elukey>	 ?
[12:38:31] <elukey>	 (one host at the time, depooling / deploying repooling every time)
[12:38:42] <elukey>	 change is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1028859
[12:39:11] <elukey>	 also for oncallers (moving the eqiad swift proxies to CFSSL/PKI, ms-fe1009 was already done yesterday)
[12:41:46] <Emperor>	 elukey: sure (I start being in meetings at 14:00 UTC but am good 'til then)
[12:42:42] <elukey>	 Emperor: okok I'll do it now then! Do you prefer to review the puppet change or shall I proceed?
[12:44:54] <effie>	 fabfur: we support feeding animals, you can go 
[12:45:22] <Emperor>	 elukey: I don't think I need to review it if you've a +1 from someone who knows about the PKI stuff :)
[12:45:55] <elukey>	 okok it was to be sure about the Swift part, perfect
[12:46:29] <klausman>	 effie: does your request extend to the ML K8ses?
[12:46:38] <effie>	 yes please
[12:47:00] <klausman>	 Alright, please lmk when I can venture into the belly of the beast again :)
[12:47:08] <effie>	 sure sure, thank you 
[12:47:43] <klausman>	 (unfortunately I just merged an admin_ng change for ML, sorry if that messes with your changes)
[12:48:32] <klausman>	 It should be constraiend to _only_ affect the ML k8ses, at least
[12:53:40] <effie>	 klausman: by accident?
[12:55:00] <klausman>	 I +2'd without thinking about Jerkins merging it
[12:55:41] <effie>	 οκ 
[12:55:43] <effie>	 ok 
[12:56:03] <effie>	 I have not merged my changes yes, so I would appreciate it if you could deploy this now 
[12:56:10] <effie>	 just to make my diff easier 
[12:56:19] <klausman>	 Will do.
[12:56:26] <effie>	 great, let me know when you are done
[12:58:22] <klausman>	 ash shucks, the change is b0rk. I'll revert my change so you can proceed.
[12:58:43] <elukey>	 I am proceeding with the swift tls upgrade, one node at the time - ms-fe1010 first
[13:00:51] <effie>	 elukey: I am terribly sorry, I had no idea you are working on this 
[13:01:12] <effie>	 elukey: can you please hold on this change? it broke maps last time 
[13:01:38] <effie>	 let me pull the task for you 
[13:02:02] <elukey>	 effie: I am already doing it, ms-fe1010 is upgraded (together with 1009 done yesterday)
[13:02:26] <elukey>	 did we attempted to migrate swift to PKI?
[13:02:30] <effie>	 yes 
[13:02:33] <klausman>	 effie: my revert is merged, from my pov you are free to proceed
[13:02:39] <effie>	 klausman: thanks 
[13:03:01] <effie>	 elukey: I am digging for the task 
[13:03:38] <effie>	 elukey: https://phabricator.wikimedia.org/T344324
[13:03:38] <elukey>	 https://phabricator.wikimedia.org/T343987
[13:04:05] <elukey>	 effie: ah wait but that is thanos right?
[13:04:11] <elukey>	 not main swift
[13:04:17] <effie>	 ah! 
[13:04:19] <effie>	 phew 
[13:04:41] <elukey>	 okok :)
[13:04:43] <effie>	 ok pebcak, sorry for raising the false alarm 
[13:06:06] <elukey>	 nono please better safe than sorry :)
[13:06:11] <elukey>	 ms-fe1010 upgraded
[13:06:14] <elukey>	 and repooled
[13:10:50] <elukey>	 proceeding with the other nodes
[13:15:40] <elukey>	 ms-fe1011 done
[13:23:16] <elukey>	 ms-fe1012 done
[13:32:02] <elukey>	 ms-fe1013 done
[13:32:05] <elukey>	 (last one standing)
[13:36:19] <elukey>	 mw-fe1014 done!
[13:36:40] <elukey>	 vgutierrez: o/ swift proxies in eqiad are all on PKI, I checked the ATS dashboard and didn't see any horror
[13:36:53] <vgutierrez>	 elukey: <3
[13:37:13] <elukey>	 will do codfw tomorrow! Cc: Emperor 
[13:37:44] <Emperor>	 thanks, looking good from here
[13:38:41] <Emperor>	 elukey: I'm starting a bit late tomorrow (ascension day, so choir things); I don't think it's a problem given you've been starting afternoon-UTC, but thought I'd mention it.
[13:38:43] <cdanis>	 elukey: <3
[13:39:32] <elukey>	 Emperor: ack thanks!
[13:39:34] <elukey>	 cdanis: <3
[14:26:39] <effie>	 Dear SRE k8s people, I finished the admin work, you may continue with your stuff
[14:28:01] * fabfur has been eaten by a tiger, too late...
[14:28:24] <effie>	 lol
[14:33:01] <klausman>	 effie: ty!
[14:57:18] <klausman>	 effie: did you perchance not push your change to ml-staging-codfw? I see a diff there but not on the ml-serve clusters.
[14:57:59] <effie>	 oh ! I missed the staging one!
[14:58:00] <klausman>	 (change as in helmfile diff)
[14:58:05] <klausman>	 no worries, I can do it
[14:58:38] <klausman>	 and done.
[14:58:38] <effie>	 klausman: can you please add this detail in https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters ?
[14:58:43] <klausman>	 Sure
[14:58:59] <effie>	 I was reading the document actually to identify where to deploy my changes 
[14:59:39] <effie>	 klausman: one last thing, after you deploy this change, you will need to delete the 2 istiod pods in the istio-system namespace 
[14:59:53] <effie>	 and let them recreate themselves
[14:59:54] <klausman>	 ack
[15:01:48] <klausman>	 istiod pods deleted and confirmed recreated