[09:36:55] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1001 for host mw2421.codfw.wmnet with OS bullseye [09:37:35] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1001 for host mw2425.codfw.wmnet with OS bullseye [09:37:38] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1001 for host mw2431.codfw.wmnet with OS bullseye [09:38:05] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1001 for host mw1472.eqiad.wmnet with OS bullseye [09:38:23] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1001 for host mw1473.eqiad.wmnet with OS bullseye [09:39:23] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1001 for host mw1474.eqiad.wmnet with OS bullseye [09:39:29] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1001 for host mw1475.eqiad.wmnet with OS bullseye [10:10:59] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1001 for host mw1473.eqiad.wmnet with OS bullseye completed: - mw1473 (**WARN*... [10:11:20] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1001 for host mw1472.eqiad.wmnet with OS bullseye completed: - mw1472 (**PASS*... [10:15:42] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1001 for host mw2425.codfw.wmnet with OS bullseye completed: - mw2425 (**WARN*... [10:16:16] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1001 for host mw1475.eqiad.wmnet with OS bullseye completed: - mw1475 (**PASS*... [10:16:47] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1001 for host mw1474.eqiad.wmnet with OS bullseye completed: - mw1474 (**PASS*... [10:20:18] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1001 for host mw2431.codfw.wmnet with OS bullseye completed: - mw2431 (**PASS*... [10:22:51] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1001 for host mw2421.codfw.wmnet with OS bullseye completed: - mw2421 (**PASS*... [11:49:55] Hi folks, could I have some envoy help, please? T351876 is a problem with swift now we've moved the proxies to use envoy rather than nginx for tls termination. v.gutierrez has suggested that envoy needs to have stream_idle_timeout set (so longer downloads still work), but AFAICT that's not obviously configurable via profile::tlsproxy::envoy hiera keys? [11:51:29] ( envoyproxy::tls_terminator does, but that looks like it's meant to be a legacy thing, and isn't how I'd made the changes ( cf T317616 )) [11:52:27] I'm afraid I don't understand envoy or its puppetry very well, but I dont' really want to try and crash-revert to nginx on a Friday :( [11:59:33] I could add a stream_idle_timeout parameter to profile::envoy::tlsproxy and pass it through? [12:02:14] [going to attempt that] [12:15:13] https://gerrit.wikimedia.org/r/c/operations/puppet/+/977178 <-- would very much appreciate a review of this, I know it's Friday, but I think swift can't be left as-is over the weekend in the light of https://phabricator.wikimedia.org/T351876 sorry :( [12:18:09] e.g. does timeout: 0.0s DTRT? [12:18:54] [stopping for lunch, will be back afterwards] [12:30:49] lol I accidentally gooled the ticket # - https://shop.deere.com/us/product/T351876%3A-Surge-Tank-Hose/p/T351876 [12:48:29] Emperor: I've +1ed your change. Looks solid to me and stream_idle_timeout is indeed 300s by default. So if the goal is to mimic nginx behaviour I would say it's fine to merge and deploy [13:02:09] thanks <3 [13:07:01] deploying [13:10:07] darn it, my test download still stopped after 85s :( [13:15:50] hm, that's odd [13:16:45] you could maybe enable debug on one of them and see what timeout actually kicks [13:18:24] Hm, trying again it's going for longer... [13:18:38] you're sure the config was already loaded? [13:18:48] it may take some time because of the hot-restarter [13:19:06] oh, that might have been it. Let's see if this download goes through... [13:19:19] [on my rubbish internet it's about 8m to download the test video from the ticket] [13:19:39] I have rubbish internet as well, I can try :-P [13:19:56] :) [13:21:17] I have friends who live in rural Cornwall with no mains water nor gas, but they get nearly 100M FTTP. I'm not jealous at all... [13:22:06] I'm done in 1m48.025s [13:22:30] but a wget with --limit-rate=1000k is also still running [13:22:53] jayme: you are not allowed to claim your internet is rubbish ;p [13:23:00] I do see that now [13:23:03] sorry :) [13:23:27] 78% done here, 2m or so to go [13:25:26] How_to_de-package_a 100%[===================>] 1.07G 1.94MB/s in 8m 44s [13:25:29] \o/ [13:25:43] (wait, do I have to watch the video now? :) ) [13:25:49] Emperor: I had better internet than that during the power outage on LTE [13:25:50] cool [13:25:52] That's... bad. [13:27:31] we only get FTTC and are a long way from the cabinet :( [13:28:01] still, T351876 is closed. Thanks for your help (and sorry for the Friday excitement) [13:28:37] sure, np [13:30:17] claime: obviously next time you have a power cut I should leave you to get the pages since you have better internet than me ;p [13:30:43] Emperor: I know you're joking, but the problem wasn't bandwidth x) [13:31:02] It cut every 10 minutes because the antennas were rebooting [13:31:11] And you can only survive for so long on powerbanks [13:31:41] details, details :) [13:31:57] Something like 90% of the mobile antennas in the département were out of order the morning of the storm [13:32:04] Insane stuff [13:33:11] swiss government _just_ (like, two months ago) slapped the telco operators with "folks, mobile network SHOULD work even if the grid is down. do something." [13:33:49] ihurbain: Some of it was because they had no power, but a good chunk was they just got destroyed or unaligned by the wind [13:34:06] We got up to 150kph inland, and 200+kph on the coast [13:35:56] see, i'm such a antenna noob i hadn't even _thought_ of alignment issues ^^; [13:36:00] coming back to the video...do people collect cleaned-off chip dies? Is that a thing? :D [13:36:22] Is it a delidding video? [13:36:37] download it yourself - you can now :-p [13:36:41] lol [13:37:15] no, the person is really stripping of the die until you can see the silicon [13:37:25] Hahaha the hot plate [13:37:28] This is so hackish [13:38:15] Oh yeah it's a complete die removal, ok apart from the debatable cool factor of having popped chips, I don't get it [13:38:29] that's why I'm asking [13:39:05] That poor die [13:39:37] well...it's a geforce 6800 ... it's probably okay ;) [13:39:49] That's fair [13:40:15] Recommended Gaming Resolutions: 640x480 - eheh [13:40:16] It's so tiny compared to modern dies lol [13:45:20] well, at least I've brought some joy to your Friday now :) [14:29:42] 10serviceops, 10Machine-Learning-Team: Bump istio Docker images to Bookworm - https://phabricator.wikimedia.org/T351933 (10elukey) [14:49:44] 10serviceops, 10Machine-Learning-Team: Bump istio Docker images to Bookworm - https://phabricator.wikimedia.org/T351933 (10elukey) Tried to build with golang 1.21 and got: ` /go/pkg/mod/github.com/lucas-clemente/quic-go@v0.28.0/internal/qtls/go120.go:6:13: cannot use "The version of quic-go you're using can't... [17:15:09] 10serviceops, 10MW-on-K8s, 10Observability-Metrics, 10SRE Observability (FY2023/2024-Q2), 10User-herron: Deploy StatsD exporter for Kubernetes - https://phabricator.wikimedia.org/T345970 (10lmata) a:05herron→03None removing Keith because he is out for a few weeks, and Service Ops is helping us with t...