[09:09:17] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10elukey) ==== Partitioning ==== As far as I can see, the kubernetes-node.cfg partman recipe creates two raid1s, one hosting root on ext4 and the... [09:09:54] hi folks, I added some ideas for the overlayfs+os upgrades in --^ [09:43:38] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10JMeybohm) For partitioning I'd prefer sticking close to the standard as well. In addition to an LV for /var/lib/docker we should probably think a... [09:52:53] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10Joe) >>! In T300744#7677738, @JMeybohm wrote: > For partitioning I'd prefer sticking close to the standard as well. In addition to an LV for /var... [09:53:07] <_joe_> elukey: I'm looking into writing the kubernetes spicerack module [09:53:45] <_joe_> at the moment I'm thinking of writing a POC using python-kubernetes first, to see if I can reproduce the behaviour of "kubectl drain" easily. [09:54:03] <_joe_> if that works, I'll follow jayme's advice and go with that [09:55:26] <_joe_> if you're both ok with it, I'll test the version on bullseye [10:02:43] sgtm [10:03:25] I'll reply to the task later/monday [10:09:21] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10JMeybohm) >>! In T300744#7677758, @Joe wrote: >>>! In T300744#7677738, @JMeybohm wrote: >> For partitioning I'd prefer sticking close to the stan... [10:51:14] _joe_: sorry I was in a meeting ,reading now [10:52:44] _joe_: +1! [11:05:37] <_joe_> elukey: you clearly have too many meetings [11:31:42] +1 [11:31:45] .) [12:55:17] <_joe_> my takeaways after two hours playing with the kubernetes api: if god wanted us to use the api, they wouldn't have given us kubectl [12:56:08] <_joe_> I have basically read the kubectl code for drain, and reproduced some of it in python. It's a lot of busywork and cross checks and sighs [13:11:12] that's uh [13:11:15] suboptimal :-D [13:11:53] _joe_: have you checked if pykube actually has an implementation for it? [14:35:54] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10elukey) Thanks for all the inputs on the partitioning, I'll try to come up with a partman recipe with all the suggestions highlighted. >>! In T3... [15:29:18] <_joe_> jayme: it just has a cordon() function which is the easy part to implement [15:31:03] 10serviceops, 10Infrastructure-Foundations, 10SRE-tools: Add a kubernetes module to spicerack - https://phabricator.wikimedia.org/T300879 (10Joe) More in detail, I would reduce the choices to a match between python-kubernetes, which we already use in imagecatalog, and kubectl. I started taking a look at how... [15:37:55] 10serviceops, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review: Many repeated config file changed / config file reloaded messages - https://phabricator.wikimedia.org/T300629 (10kostajh) 05Open→03In progress [15:40:30] _joe_: hm, yeah. cordon just sets spec.unschedulable I'd guess [15:40:43] <_joe_> yes [15:40:45] <_joe_> question: do we disable the discovery api on our kubemasters? [15:41:38] you mean some openapi endpoint? [15:41:53] <_joe_> I know that's at /openapi/v2 [15:42:19] <_joe_> but I have some code failing in my poc with python-kubernetes [15:43:40] I'm not aware that we disable it on purpose fwiw [15:46:23] <_joe_> https://phabricator.wikimedia.org/P20183 [15:46:33] <_joe_> yeah I wouldn't think we do [15:46:50] <_joe_> basically that's the interface that tells you which api groups the server supports [15:47:08] <_joe_> and it's how kubectl decides if it should evict pods or delete them [15:47:28] ah, because eviction api was late to the party [15:47:36] <_joe_> yeah [15:47:45] <_joe_> I think we can ignore it but it made me curious [15:52:19] hm, no idea tbh [17:35:03] _joe_: you still arround? [17:35:18] <_joe_> yes [17:36:29] nice :) I'm looking for the place where the service::catalog "monitors" stuff gets transformed to pybal config [17:37:02] <_joe_> I guess in the erb file for the pybal configuration [17:38:03] ah, damn...so easy [17:38:06] <_joe_> jayme: https://github.com/wikimedia/puppet/blob/production/modules/pybal/templates/pybal.conf.erb#L65 [17:38:09] sorry for the noise [17:38:49] I actually thought it is in etcd as well and did not suspect find a actual config file [17:44:29] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10Volans)