[09:17:08] 10serviceops, 10Campaign-Tools, 10MW-on-K8s, 10Patch-For-Review: Setup sendmail on k8s container - https://phabricator.wikimedia.org/T325131 (10Joe) >>! In T325131#8471194, @jhathaway wrote: >>>! In T325131#8470355, @Joe wrote: >> My current idea for attacking this is the following: >> * install msmtp and... [10:16:27] 10serviceops, 10Platform Engineering, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Phase out "redis_sessions" cluster and away from memcached cluster - https://phabricator.wikimedia.org/T267581 (10jijiki) 05Open→03Resolved [10:31:17] hi folks! [10:32:15] I am reviewing the sre.k8s.pool-depool-cluster cookbook, and I noticed that kubesvc is not really touched [10:32:30] is there a specific reason or could it be a feature to add? [10:33:05] (like: by default we don't touch it, but with a new option one can also depool services from kubesvc) [11:06:50] elukey: you mean the nodes are not set to inactive or something in confctl? [11:07:38] jayme: yes exactly [11:08:30] elukey: IIRC the scope of the coobook was just to depool all the services running on the cluster, not the cluster itself (I get that the name might be misleading) [11:09:06] 10serviceops, 10GitLab, 10serviceops-collab, 10Patch-For-Review: Trusted gitlab runner containers need access to kubestagemaster k8s cluster - https://phabricator.wikimedia.org/T325385 (10Jelto) Thanks for opening the task! As far as I know direct access from CI workers to the Kubernetes API is a new feat... [11:10:47] 10serviceops, 10GitLab, 10serviceops-collab, 10Kubernetes, 10Patch-For-Review: Trusted gitlab runner containers need access to staging k8s cluster - https://phabricator.wikimedia.org/T325385 (10taavi) [11:10:54] 10serviceops, 10GitLab, 10serviceops-collab, 10Kubernetes, 10Patch-For-Review: Trusted gitlab runner containers need access to staging k8s cluster - https://phabricator.wikimedia.org/T325385 (10taavi) [11:11:44] jayme: I am missing something then. IIUC "service: kubesvc" in service.yaml as used before the Istio ingress solution to manage service traffic. Are we using only the istio ingress on wikikube? And what do you mean with "not the cluster itself" ? [11:14:32] s/as/was [11:25:55] 10serviceops, 10Campaign-Tools, 10MW-on-K8s, 10Patch-For-Review: Setup sendmail on k8s container - https://phabricator.wikimedia.org/T325131 (10Joe) Correcting myself: ugh, ssmtp's source code has last been updated about a decade ago, it's not really a viable alternative. I think what I have prepared is e... [11:29:51] elukey: hmm...quick meet maybe? [11:30:15] jayme: need to go in a bit, do you have time this afternoon? [11:31:35] elukey: yes. Between 14.30Z and 16.00Z [11:41:21] 10serviceops, 10Campaign-Tools, 10MW-on-K8s, 10Patch-For-Review: Setup sendmail on k8s container - https://phabricator.wikimedia.org/T325131 (10MatthewVernon) Is it worth patching msmtp to make it a bit more ... conventional? Or is that too much hassle? [11:48:53] 10serviceops, 10Campaign-Tools, 10MW-on-K8s, 10Patch-For-Review: Setup sendmail on k8s container - https://phabricator.wikimedia.org/T325131 (10Joe) >>! In T325131#8477871, @MatthewVernon wrote: > Is it worth patching msmtp to make it a bit more ... conventional? Or is that too much hassle? It's both too... [12:01:39] <_joe_> elukey: kubesvc is just a conftool label,and it mostly pertains to pybal's configuration, don't fixate on it [12:02:07] <_joe_> depooling a cluster here means moving all the dns discovery records for everything running on it away from it [12:02:18] <_joe_> which means that in theory no traffic will reach pybal [14:20:02] _joe_ yes sure I am not fixating, it is just a matter of adding some comments/clarity here and there in cookbooks so others will know as well [14:20:33] one of the points of the maintenance cookbook is to think about depooling or not kubesvc, or just silencing pybal etc.. [14:20:45] I'd like to understand pros/cons and add some comments so it is clear [15:48:55] 10serviceops, 10GitLab, 10serviceops-collab, 10Kubernetes, 10Patch-For-Review: Trusted gitlab runner containers need access to staging k8s cluster - https://phabricator.wikimedia.org/T325385 (10dancy) [16:01:13] 10serviceops, 10GitLab, 10serviceops-collab, 10Kubernetes, 10Patch-For-Review: Trusted gitlab runner containers need access to staging k8s cluster - https://phabricator.wikimedia.org/T325385 (10dancy) [16:02:53] 10serviceops, 10GitLab, 10serviceops-collab, 10Kubernetes, 10Patch-For-Review: Trusted gitlab runner containers need access to staging k8s cluster - https://phabricator.wikimedia.org/T325385 (10dancy) >>! In T325385#8477814, @Jelto wrote: > As far as I know direct access from CI workers to the Kubernetes... [16:05:24] 10serviceops, 10Campaign-Tools, 10MW-on-K8s: Setup sendmail on k8s container - https://phabricator.wikimedia.org/T325131 (10jhathaway) >>! In T325131#8477891, @Joe wrote: >>>! In T325131#8477871, @MatthewVernon wrote: >> Is it worth patching msmtp to make it a bit more ... conventional? Or is that too much h... [16:13:55] 10serviceops, 10Campaign-Tools, 10MW-on-K8s: Setup sendmail on k8s container - https://phabricator.wikimedia.org/T325131 (10Joe) >>! In T325131#8478705, @jhathaway wrote: >>>! In T325131#8477891, @Joe wrote: >>>>! In T325131#8477871, @MatthewVernon wrote: >>> Is it worth patching msmtp to make it a bit more... [16:16:45] 10serviceops, 10Campaign-Tools, 10MW-on-K8s: Setup sendmail on k8s container - https://phabricator.wikimedia.org/T325131 (10jhathaway) You could always run systemd in your container 😉. Possible jokes aside, I think msmtp is fine short term, while I work to get an LVS backed outgoing mail service up and running. [16:25:09] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10thcipriani) [18:06:08] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 3 others: Write a cookbook to set a k8s cluster in maintenance mode - https://phabricator.wikimedia.org/T277677 (10elukey) I had a chat with Janis, and this is what I am going to do: 1) Refactor where possible `re.k8s.pool-depool-cl... [18:07:30] 10serviceops, 10Campaign-Tools, 10MW-on-K8s: Setup sendmail on k8s container - https://phabricator.wikimedia.org/T325131 (10Legoktm) >>! In T325131#8469178, @jhathaway wrote: > Longer term I think it would be better to use another mail library and send via SMTP directly to our outgoing mail servers. I was co... [19:37:22] 10serviceops, 10Campaign-Tools, 10MW-on-K8s: Setup sendmail on k8s container - https://phabricator.wikimedia.org/T325131 (10jhathaway) >>! In T325131#8479122, @Legoktm wrote: > Since it hasn't been mentioned yet, to be clear, MediaWiki already supports SMTP directly: https://www.mediawiki.org/wiki/Manual:$wg... [20:06:07] 10serviceops, 10SRE: service implementation tracking: arclamp1001.eqiad.wmnet - https://phabricator.wikimedia.org/T319434 (10aaron) >>! In T319434#8383918, @Dzahn wrote: > per T316223#8381863 serviceops-core is taking this over Let us know if there is anything you need from the perf team. [23:14:41] 10serviceops, 10SRE: service implementation tracking: arclamp1001.eqiad.wmnet - https://phabricator.wikimedia.org/T319434 (10Dzahn) @aaron CCing @akosiaris Depending how you want to look at it this is either a subtask and unblocks or a duplicate of T316223. Also see T316223#8383941, T316223#8185277. Cheers