[00:57:17] has anyone noticed as well when you use rsync with stunnel the file transfer itself works but then the connection is not ended properly and it just sits there until TIMEOUTclose exceeded: closing? like the final FIN is not being received, or rsync does not properly close stunnel? [00:57:57] maybe it is happening all the time but we don't care because the rsync still happened and the systemd unit is not marked as failed [00:58:46] and maybe I don't have to care because the part that matters works [01:15:28] yea, I think it's the latter. we dont care, it's just noise/warning [08:33:10] moritzm: looks like puppet run is stuck on ===> Starting run on puppetmaster2002.codfw.wmnet... [08:33:30] looking at https://phabricator.wikimedia.org/rOPUPb0a636562aecdadc9cd368efe499f56e5cc9c0a6 you're working on decom it? [08:34:17] finally hit the timeout, maybe we should lower it? [08:34:22] jelto: ^ [08:34:36] from where is the output? [08:34:54] puppetserver1001:~$ sudo -i puppet-merge [08:34:55] puppet-merge [08:34:55] but yes, puppetmaster2002 is currently being shut down, I had applied the insetup::buster role to it [08:35:54] hmmh, puppet-merge syncs based on the role name, so with the change of the role that should no longer happen? [08:36:08] maybe for next run? [08:37:14] I guess so [08:39:52] With Jelto we've just merged a CR written by volans in 2020 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/618766 :) [08:40:56] so, I had a closer look: the workers are written to /etc/puppet-merge/shell_config.conf, which is a Puppet template, so I ran puppet on puppetserver1001 and now puppetmaster2002 is gone from there and should no longer be synced [08:41:10] perfect [08:42:12] lol, didn't the patch went bad/expired? :-P [08:42:33] not even! [08:44:46] :D [08:53:45] moritzm: do you know if we have "extlib" enabled ? https://forge.puppet.com/modules/puppet/extlib/reference#extlibnetmask_to_cidr [08:56:49] no, this gets provided by puppet-module-extlib, which we currently don't have installed [08:57:40] https://tracker.debian.org/pkg/puppet-module-extlib [08:57:50] ok, thx! [10:22:05] XioNoX: we have an equivalent as wmflib::mask2cidr [10:23:13] yeah that's what we ended up using, but was wondering if there was a more official one first [12:28:50] Did something change with how logstash.wikimedia.org works in relation to request IDs? It seems WikimediaDebug is going into a recursive tailspin as of a few days ago, where seemingly it thinks it is instrumenting logstash.wikimedia.org itself. [12:29:19] Which means when you debug en.wikipedia.org with Verbose logs and click the logstash menu, WikimediaDebug then fills up with hundreds of logstash.wikimedia.org requests that it thinks it has debugged. [12:30:07] I'm guessing it has gained a new header or changed exposure to some vcl/ats rule that adds or no longer strips a x-request-id header? [12:53:58] Krinkle: There are a few weird (possibly not related) things happening with logstash right now [12:54:06] Like all times seem to be US WC time [12:58:21] (took me a second to parse "WC" as "west coast" :D ) [12:59:10] * Krinkle read it as bathroom time. [12:59:18] Ah yeah, I noticeed that in Lisbon last week too, yeah. [12:59:49] ihurbain: yeah I couldn't remember the timezone name lmao [12:59:59] Krinkle: same. "that's a weird place to look at logstash, but whatever works, i guess" [13:00:10] also west coast is always typed in capitals like you're hyping up a rapper WEST COAST [16:45:17] Krinkle: ... ah. probably because of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1194989 [16:45:57] workaround suggestions appreciated 😅 [19:19:56] cdanis: hm.. why expose them to the web client for non-xwd? [19:20:11] (changing that would not solve the issue, I think, just curious) [23:25:20] Apropos nothing at all (Codesearch... :whistle:) [23:25:28] "How, in the year 2025, is inode exhaustion still a thing which happens" https://chaos.social/@russss/115730690838781310 [23:26:12] They're hosting map tiles.