[11:21:44] 10serviceops, 10envoy, 10Patch-For-Review, 10SRE Observability (FY2021/2022-Q3), 10User-fgiunchedi: Using port in Host header for thanos-swift / thanos-query breaks vhost selection - https://phabricator.wikimedia.org/T300119 (10Joe) For the record, what happened here is as follows: * envoy had thanos-qu... [11:44:03] 10serviceops, 10envoy, 10Patch-For-Review, 10SRE Observability (FY2021/2022-Q3), 10User-fgiunchedi: Using port in Host header for thanos-swift / thanos-query breaks vhost selection - https://phabricator.wikimedia.org/T300119 (10Joe) Results after merging the patch on thanos-fe1001 solve our immediate pro... [12:00:05] 10serviceops, 10envoy, 10Patch-For-Review, 10SRE Observability (FY2021/2022-Q3), 10User-fgiunchedi: Using port in Host header for thanos-swift / thanos-query breaks vhost selection - https://phabricator.wikimedia.org/T300119 (10Joe) Given the "expected" if not necessarily correct behaviour has been intro... [12:01:28] 10serviceops, 10SRE, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10Joe) Thanks @rzl this looks like an excellent plan. I would suggest that when we move to 1.18, we might want to start from the `thanos-fe` cluster which would see fixing of a real iss... [12:11:33] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [12:14:51] 10serviceops, 10SRE, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10Vgutierrez) this looks great :) in traffic we're already using 1.18.3 from the envoy-future component, thanks @RLazarus [13:17:03] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Kubernetes services with externalTrafficPolicy: Local don't work - https://phabricator.wikimedia.org/T300500 (10JMeybohm) 05Open→03Resolved [14:14:32] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Provide a convenient way to connect to services in kubernetes staging clusters - https://phabricator.wikimedia.org/T300740 (10JMeybohm) [14:21:33] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [15:31:32] 10serviceops, 10Machine-Learning-Team: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10elukey) [15:58:57] 10serviceops, 10SRE, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10Joe) >>! In T300324#7670904, @Vgutierrez wrote: > this looks great :) in traffic we're already using 1.18.3 from the envoy-future component, thanks @RLazarus I think the question we... [16:05:23] 10serviceops, 10SRE, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10hnowlan) Based on the release notes I think the API gateway will most likely have no issue going straight to 1.21. If there are issues they will most likely be minor enough that we can... [16:24:53] 10serviceops, 10Machine-Learning-Team: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10elukey) [17:42:15] hello friends. just wondering about a temperature check on an idea that is older than I am and even less pleasant - restbase on k8s. Any thoughts? We're looking at moving it to node 12 and I'd rather just bite this particular bullet than do another wave of reimages right off the bat. [17:42:39] restbase will be going away at some point but it has been about to go away for years [17:47:50] I remember we had talked about that at the last sync, right? IIRC the base line was that it's seen as pretty pleasant (the end result, not the way to it ofc :)) [17:48:53] one of the main concerns is the dht ratelimiter built into it but I am fairly sure that it doesn't work and might not have worked for a very long time (https://phabricator.wikimedia.org/T249699) [18:00:50] but what's the concern? The UDP traffic in general? /should/ work I guess :) [18:02:33] and if you're not going to re-enable it anyways it seems even more safe [18:04:36] changed behaviour I guess is the main thing (if it were otherwise somehow working, heh) [18:05:31] If it were in theory working the protocol is gossip-based so that would most likely have issues in k8s, but we can remove or change it even if needed [18:12:18] I think we could even get it to work in k8s if we wanted to. But I've not checked in detail ofc [18:13:07] as long as it does not rely on static IPs (and can use semi or fully static hostnames) it sounds doable in k8s [18:22:07] sounds good [18:22:40] I'm still inclined to see what happens if we turn it off in prod first to see if we take any kind of real hit [18:27:07] yeah. Sounds reasonable to seperate that from the nodejs upgrade / k8s migration [21:51:21] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.3.0 - https://phabricator.wikimedia.org/T300804 (10dancy) [22:41:18] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) Alright! the TLS cert issue should be fixed with the above ^ at least in this sense: ` Feb 02 22:36:55 gitlab-prod-1001 systemd[1]: certbot.service: Suc... [22:43:56] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [22:44:40] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [22:46:10] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) this is not puppet-related anymore. Manually installing gitlab-ce also does this: ` Checking PostgreSQL executables: OK Checking if a newer PostgreSQL... [22:52:36] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [23:24:26] 10serviceops, 10SRE, 10GitLab (Infrastructure): gitlab: enable IPv6 for https - https://phabricator.wikimedia.org/T300816 (10Dzahn)