[09:17:09] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10awight) I like this approach very much. Can you share thoughts about how we'll... [09:23:53] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Ladsgroup) >>! In T308932#7955791, @awight wrote: > I like this approach very mu... [10:22:22] Just wanted to be sure around language on the ingress docs - does this section more or less mean that you can/should go straight from service_setup to production? https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress#Create_an_entry_in_the_service::catalog [10:22:45] Or should I do the other transitions in order even though they have no impact [12:17:11] <_joe_> hnowlan: I'll take a look in a bit, as the author of that page isn't available today [12:17:54] <_joe_> but yes it makes sense to go from service_setup to monitoring_setup or production, once you've deployed your service [12:20:20] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Jdforrester-WMF) C-2 I think this split is a very bad idea. Our configuration i... [12:39:40] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Ladsgroup) While your point is valid, I think it's about a different aspect than... [12:42:12] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Jdforrester-WMF) >>! In T308932#7956428, @Ladsgroup wrote: > While your point is... [12:46:49] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Ladsgroup) >>! In T308932#7956429, @Jdforrester-WMF wrote: >>>! In T308932#79564... [13:08:30] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Jdforrester-WMF) >>! In T308932#7956503, @Ladsgroup wrote: >>>! In T308932#79564... [13:45:20] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Ladsgroup) Having a system in noc.wikimedia.org for users to be able to see conf... [13:50:52] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Jdforrester-WMF) >>! In T308932#7957024, @Ladsgroup wrote: > Having a system in... [15:16:03] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2022: Iteratively clean up wmf-config to be less dynamic and with smaller settings files (2022) - https://phabricator.wikimedia.org/T308932 (10Ladsgroup) This is the POC ^. Obviously it has the downside of not showing confi... [15:16:13] does the service-runner provide some mechanism for layering configuration files? [15:16:40] i want to override a few config settings for a particular environment (beta cluster) without maintaining a duplicate configuration file [15:32:39] <_joe_> ori: I have no idea, but hnowlan or urandom would possibly know [16:04:39] ori: not that I know of, no (but I'm hardly an expert). [16:05:53] sadly no one in the list of "experts" is...handy, anymore :( [16:06:33] cc: @Nikerabbit (?) [16:07:41] urandom: if this is about service-runner, I think you got the one person [16:10:45] yeah, sorry, typo! [16:10:53] ori: fwiw we use helm templates with differing variable files to override options for beta in changeprop but that's still a single config file [16:11:18] cc: nikkinikk ^^^ [16:14:17] <_joe_> doesn't look like adding the ability to do deep merges of configuration should be impossible [16:15:06] <_joe_> ori: now just write a design doc, submit it to the relevant annual plan committee, work with project managers to transform it into a OKR, then by 2024 or 2025 it should be available. [16:15:26] <_joe_> or you know, send a patch :P [16:15:52] I'll file a bug at minimum [19:56:29] Does anyone have time to help me figure out the ingress for the new developer-portal deployment in k8s-staging? I believe all the things from https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress#Configuration_(for_service_owners) have been done, but I'm not able to get https://developer-portal.staging.discovery.wmnet, https://developer-portal.k8s-staging.discovery.wmnet, or curl -v https://staging.svc.eqiad.wmnet:4016 to work. [20:07:46] bd808: this seems to work: `curl https://developer-portal.k8s-staging.discovery.wmnet:30443` [20:08:50] discovered that based on https://phabricator.wikimedia.org/T300740#7828099 [20:09:22] taavi: cool! I was going to ask where that port number comes from, but you gave your citation :) [20:10:05] * bd808 will make some edits on https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress [20:11:27] https://integration.wikimedia.org/ci/job/helm-lint/7413/console somehow claims it'd use default.k8s-staging.discovery.wmnet.. I wonder if that's a bug with the diff job or an actual problem [20:11:45] 'Error from server (Forbidden): destinationrules.networking.istio.io is forbidden: User "developer-portal" cannot list resource "destinationrules" in API group "networking.istio.io" in the namespace "developer-portal"' makes verifying that unfortunately hard [20:14:43] The output from staging looks good. I'm going to try deploying into codfw now. [20:25:15] The codfw deployment is failing to roll out with an error of "Failed to load certificate chain from /etc/envoy/ssl/service.crt" from the envoy sidecar. I had to login with a second shell to figure this out as the `helm` command is just sitting there with no indication of what is happening under the hood. [20:25:44] * bd808 tries to figure out what is wrong with this part of the chart [20:27:36] the certificate contents in that config map are 'snakeoil' apparently :/ [20:27:58] * taavi though that was all handled by cfssl [20:32:05] taavi: I guess we need https://wikitech.wikimedia.org/wiki/Kubernetes/Enabling_TLS#Create_and_place_certificates then. https://wikitech.wikimedia.org/wiki/Kubernetes/Add_a_new_service#Create_certificates_(for_the_services_proxy) says that is needed for both ingress and LVS setups. [20:33:03] sounds likely. [20:40:42] 10serviceops, 10Wikimedia-Developer-Portal, 10Goal, 10Service-deployment-requests: New Service Request: developer-portal - https://phabricator.wikimedia.org/T297140 (10bd808) Deployment into the "staging" environment worked: ` $ curl -I https://developer-portal.k8s-staging.discovery.wmnet:30443 HTTP/2 200... [20:40:48] 10serviceops, 10Wikimedia-Developer-Portal, 10Goal, 10Service-deployment-requests: New Service Request: developer-portal - https://phabricator.wikimedia.org/T297140 (10bd808) Deployment into the "codfw" cluster is failing with the pod in CrashLoopBackoff. The error is: ` $ kubectl logs developer-portal-ma... [21:37:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q4: (Need By: TBD) rack/setup/install mw14[57-98] - https://phabricator.wikimedia.org/T306121 (10Jclark-ctr) [21:46:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q4: (Need By: TBD) rack/setup/install mw14[57-98] - https://phabricator.wikimedia.org/T306121 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson [22:05:45] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 (10brennen) To save splitting discussion here,... [22:06:08] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 (10brennen) 05Stalled→03Resolved a:03bren... [22:47:25] 10serviceops, 10Wikimedia-Developer-Portal, 10Goal, 10Service-deployment-requests: New Service Request: developer-portal - https://phabricator.wikimedia.org/T297140 (10RKemper) Followed the steps in https://wikitech.wikimedia.org/wiki/Kubernetes/Enabling_TLS#Create_and_place_certificates with @bking, @bd80...