[09:45:27] 10serviceops, 10Release-Engineering-Team, 10serviceops-collab, 10GitLab (CI & Job Runners), and 2 others: Changes to modules/gitlab_runner/templates/config-template.toml.erb have no effect on existing runners - https://phabricator.wikimedia.org/T311746 (10Jelto) 05Open→03Resolved >>! In T311746#8107945... [13:11:41] 10serviceops, 10Page Content Service, 10Product-Infrastructure-Team-Backlog: Production configuration for mobileapps need to be adapted to codechanges - https://phabricator.wikimedia.org/T320505 (10Jgiannelos) [14:00:21] Hi! I sent a deployment-charts patch for T320505, not sure who to add for reviews: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/841502/ [14:08:41] nemo-yiannis: left a comment [14:09:00] thanks! [14:29:10] 10serviceops, 10Discovery-Search, 10SRE, 10serviceops-collab, and 2 others: Sunset search.wikimedia.org service - https://phabricator.wikimedia.org/T316296 (10LSobanski) a:03Dzahn [14:30:27] 10serviceops, 10Observability-Logging, 10Observability-Metrics, 10SRE, 10Performance-Team (Radar): Framework for running experiments on a subset of the app server fleet - https://phabricator.wikimedia.org/T315403 (10LSobanski) [14:46:30] 10serviceops, 10serviceops-collab: convert misc services with multiple backends to active-active where possible - https://phabricator.wikimedia.org/T263506 (10LSobanski) a:05Dzahn→03None [15:03:02] effie: inb4 train emoji gets changed to baby emoji [15:04:29] 10serviceops, 10Infrastructure-Foundations, 10Patch-For-Review: Make gVisor packages available via apt.wikimedia.org - https://phabricator.wikimedia.org/T316879 (10ori) @Joe the Wikifunctions Beta Cluster instance is running Bullseye -- could you also pull it in there? [15:12:12] 10serviceops, 10Infrastructure-Foundations, 10Patch-For-Review: Make gVisor packages available via apt.wikimedia.org - https://phabricator.wikimedia.org/T316879 (10ori) Never mind, I see that it is available for Bullseye -- sorry. [15:12:14] I will find an emoji, for syre [15:12:16] sure* [15:21:23] 10serviceops, 10Kubernetes: Integrate kube-metrics-server into our infrastructure - https://phabricator.wikimedia.org/T249929 (10BTullis) I suggest that we merge {T318925 } in as a duplicate of this ticket. [15:49:25] ahoyhoy - effie and I were chatting about cutover plans for thumbor-on-k8s and the question of how to do a mixed environment of metal and k8s instances came up. Currently rewrite.py is configured to use the lvs service, which itself gets its hosts via conftool. Could we add the k8s nodes to that conftool pool and manage the weights to gradually introduce the k8s hosts or would that be messy? [15:50:26] Or would it make more sense to add logic to rewrite.py (the swift script that sends requests to thumbor, to clarify) to support balancing requests between multiple hosts and control it that way, using a new lvs endpoint for the thumbor-k8s hosts? [15:53:03] (I can move this to a phab ticket if needs be, I realise this might be a bit of a longer convo) [15:56:47] <_joe_> hnowlan: in the past we just went with adding k8s nodes to lvs progressively [15:56:58] <_joe_> but in this case, there's an added complexity [15:58:17] <_joe_> uhm, no, thumbor listens on port 9900 (via haproxy), right? [15:58:43] <_joe_> so I guess we can add a few k8s hosts with weight << than the current weight [15:59:55] 8800 afair [16:00:41] but yeah, makes sense. would we have to add all k8s hosts? [17:36:05] lunch/errands, back in ~1h