[01:40:27] mutante: thanks! it was fun [07:12:05] the mailman queue has recovered [08:11:08] I am wondering why it apparently uses a single process to emit emails [08:24:16] https://www.openssh.com/releasenotes.html [08:24:22] * ssh(1): add a new EnableEscapeCommandline ssh_config(5) option that [08:24:22] controls whether the client-side ~C escape sequence that provides a [08:24:22] command-line is available. Among other things, the ~C command-line [08:24:22] could be used to add additional port-forwards at runtime. [08:24:32] This option defaults to "no", disabling the ~C command-line that [08:24:32] was previously enabled by default. [08:24:39] this is going to catch people by surprise ... [08:25:03] I just caught it by reading changelogs while upgrading to bookworm [08:25:47] Turning off the command-line [08:25:47] allows platforms that support sandboxing of the ssh(1) client [08:25:47] (currently only OpenBSD) to use a stricter default sandbox policy. [08:26:20] ok, so only 1 OS supports that, but let's break workflows for who knows how many people to allow that 1 OS to use a stricter policy [12:23:02] The sre.puppet.sync-netbox-hiera cookbook wants to add data for urldownloader1004 [12:23:13] (running as normal after dns cookbook) [12:23:22] I'm gonna go ahead and let it proceed [12:52:11] <_joe_> There are ongoing issues with file uploads to commons (see wikimedia-l, and my ping in this channel yesterday). I am off today but you should really investigate. On first sight of logstash, it seemed like the issue was with swift [14:03:08] hnowlan: nearly all the 50Xs I'm seeing in swift logs relate to thumbs; is it possible thumbor is sad again? [14:18:19] akosiaris: I have done https://phabricator.wikimedia.org/T331508 can you confirm everything still working fine? :) [14:48:13] it does concern me a bit that both eqiad and codfw 50x rates both bump up noticeably at almost exactly 13:00 yesterday (when AFAICT no relevant change seems to have taken place), seems maybe thumbor-sadness [14:51:00] https://www.troyhunt.com/to-infinity-and-beyond-with-cloudflare-cache-reserve/ [14:51:14] Essentially boils down to, adding Swift behind Varnish :) [15:05:06] marostegui: well, https://toolhub.wikimedia.org/ is working and also it is using the next set of IPs since Tuesday. I guess we would heard if it was broken by now [15:05:46] akosiaris: with mysql grants you never know what surprise you might be facing [15:07:43] marostegui: isn't it routed via a proxy? so the direct grants would not be used anyways [15:08:26] taavi: Everything going to misc should go thru proxies, but we've found legacy issues in the past messing up with the proxies, or hardcore hostnames [15:08:31] That's why I prefer to ask anyways :) [16:51:04] * Emperor no nearer any answers on the swift error rate increase [16:51:27] swift logs seem unremarkable, thumbor error rates don't seem to have changed noticeably (or I've just picked the wrong things to look at) [16:51:41] have added some notes to T328872 [16:51:41] T328872: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw - https://phabricator.wikimedia.org/T328872 [17:06:54] (not least, ATS is logging 502s and swift logs basically only contain 500s, so I'm not even entirely sure where the errors are coming from)