[13:43:02] on-callers: (unless there are objections) I will momentarily begin a rolling reboot of sessionstore (no impact expected) [14:43:53] late to the game, but no objections urandom [14:44:31] jhathaway: that's good, because they are almost done. :) [14:44:46] :), I figured as much! [15:30:39] dancy (or whoever), can I get a quick opinion about whether these pipeline failures are a me problem or a you problem? https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/jobs/841896 [15:31:31] Checking [15:31:43] huh, that was not the channel I meant to write that in. But thanks! [15:32:23] Is this the program that generates a huge image? [15:34:48] yes it is [15:35:09] It had a good run of not failing for several months though [15:57:16] andrewbogott: Root cause discovered: There's a 5 minute transfer timeout in play that didn't exist a while ago (before we switched from ingress-nginx to Traefik). Before simply increasing the timeout, I'm going to look at your job a bit more to see if it is even useful for it to be exporting its cache. If not, I'll add a way to disable that action. [15:58:00] thank you for looking [15:58:07] I have no real agenda, whatever works sounds good to me [16:21:32] hi, i wanted to ask, how frequently envoy services_proxy config (e.g. https://w.wiki/QVfP) is rolled out to the wiki k8s? i'm planning an mw-config change that will depend on the ports introduced in services_proxy change, when will it be safe to backport? [16:42:29] atsukoito: as soon as that patch is merged, puppet-merged, and after puppet agent runs on the deploy host being used [16:43:54] cdanis: this i understand, but it'll be only updated on deploy node, what about having this port on the actual pod? [16:44:13] puppet maintains local files there that are read by helm on every deploy :) [16:44:31] the generated envoy config (or at least, the relevant parts of it) get injected into a configmap [16:46:13] when this configmap is going to be regenerated? when backporting with spiderpig? when applying admin-ng? i still can't figure this out [16:46:32] when backporting with spiderpig [16:46:48] scap is what invokes helmfile [16:46:54] i see, thanks [16:47:16] soon we'll have https://wikitech.wikimedia.org/wiki/Sophroid [16:51:25] can i ask for +1 for that diff, btw? most of my team is off already https://gerrit.wikimedia.org/r/c/operations/puppet/+/1295901 [16:51:40] ah, nvm, brouberol just did it [16:53:04] atsukoito: when merged, run puppet on the deployment server. This will update /etc/helmfile-defaults/dse-k8s-eqiad-general.yaml, which is read by all our helmfiles. Meaning that you'll be able to redeploy whatever depends on your changes [17:14:22] thnx!