[06:46:27] FWIW, I got Amir.1's notification in German too (and haven't set German as a language anywhere) [13:15:36] Google calendar/slides/... messages being in the language of the sender rather than recipient is a... well, I wouldn't call it a known issue, but rather a known-bad-choice :) [13:17:31] One of the problems is of course that the system may just not know the language of the recipient (external email) and guessing it would only make things worse. Whether it should be able to know the language of recipients that are Goo customers has been debated extensively [14:17:16] mutante: I assume https://phabricator.wikimedia.org/T114801 (operations-apache-config-lint replacement doesn't check syntax) is no longer relevant right? Do you know if we check syntax pre-merge in puppet nowdays through other means? [14:17:56] or whether that's still something we want? vs checking e.g. in beta or prod on a depooled host via httpbb [14:47:05] Krinkle: I think it's still relevant, it's just that we don't have as manu apache changes anymore, and rewrite rules are generated [14:47:10] many [14:48:20] Krinkle: the check is usually do disable puppet on mw*, deploy, enable puppet, check on mwdebug with httpbb. but not in CI or beta [14:48:46] if it's something new you'd first have to add tests for httpbb [16:01:17] _joe_: `[merged] /wmf-config/redis.php -18B Deleted` [16:01:27] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/824737 [16:02:27] Question about swiftrings and rsync::server module, should a host that is not ring_manager have a running rsync daemon? [16:06:47] * claime in the puppet weeds [16:09:24] claime: I'm about to vanish on holiday, but T311066 is the bug I opened about this [16:09:24] T311066: rsync::server::module installs an rsync server even when $ensure is absent - https://phabricator.wikimedia.org/T311066 [16:09:49] Emperor: welp I stumbled on it while looking for something to do with the puppet runs lol [16:09:50] claime: so TL;DR> no, they shouldn't, but rsync::server::module makes that impossible right now. [16:10:00] Emperor: Because of the include, right? [16:10:16] Yeah, reading the phab bug [16:10:26] That's exactly what I ended up at [16:11:54] rsync::server::module has an ensure parameter, set it to "absent" based on "is not ring_manager" (hiera setting?) [16:11:56] I got a layer of sidetracking because of a command for fragments that wasn't run correctly [16:13:02] mutante: as noted in the phab item, doing that still ends up with an (unconfigured) rsync server installed. [16:13:02] Yeah that may work, set it to absent with a big conditional but... ugly [16:13:27] Anyhow, sorry, I have to go get ready to catch a train. [16:13:31] Emperor: wrapping the include in a conditional based on the ensure *should* work but, again, ugly. [16:13:33] that kind of conditional is fairly standard [16:13:35] Emperor: no worries [16:13:53] patches would be a welcome thing to return to after my holiday :-D [16:13:59] we do this all the time to set an $ensure to present/absent based on Hiera ('active server' etc) [16:14:13] ACK @ the bug that it still leaves an rsyncd [16:14:16] Also, I doubt that will uninstall the currently unconfed rsync server, but that can be cleaned up manually [16:14:24] but without a fragment it won't be usable [16:15:09] mutante: ok so my path through the bug was: fragment doesn't exist and command to set to header only is buggy https://gerrit.wikimedia.org/r/c/operations/puppet/+/825793 [16:15:12] I would not even try to manually clean up [16:15:20] that usually leads to more issues [16:15:25] So systemd doesn't want to start rsync, but puppet tries every time [16:16:00] Fixing the command would create the file on header update, leading to a working rsyncd, but it would be unconfigured [16:16:04] this must be a new bug [16:16:13] or something else fixable in puppet [16:16:39] The command is quite old in the code, but it could never work [16:16:51] we have this situation all over the place where rsyncd is included in a role though [16:17:17] it's like a standard thing. and if rsyncd fails to start then it should Icinga alert on all of them because of systemd status [16:17:41] can you point me to the code where the rsync server is used/included? [16:18:12] modules/rsync/manifests/server/module.pp+59 [16:18:21] is the include [16:19:01] same file line 91 is the command that is supposed to create a header-only rsync conf but doesn't [16:19:07] I meant where you are actually using the rsync module [16:20:58] It's in modules/profile/manifests/thanos/swift/frontend.pp through swift::ring_manager which is modules/swift/manifests/ring_manager.pp line 70 [16:21:08] Sorry, had to backtrack in my editor a bit lol [16:25:06] claime: gotcha, so it already uses "ensure => $ensure". on which host do you get the error for example? [16:25:25] thanos-fe2002.codfw.wmnet is an example [16:28:15] But I would not be surprised if a big chunk of the 37 hosts we have that have puppet changes at every run were from this situation, there's at least the ms- servers, plus the thanos- and probably most others that use the swift::ringmanager because of the included rsync::server having a default ensure_service at running [16:29:50] mutante: from what I gather, the include is done this way so you can have multiple modules, which is fine in itself, but if we want to ensure_service stopped, then we need a way to make sure the module with ensure absent is the only one on the server [16:29:54] does that make sense? [16:31:18] yea, if there are 2 modules with conflicting $ensure that would explain flapping / change on every run [16:31:41] can't you just set $ensure to absent on both though [16:32:24] there are plenty of hosts with multiple rsync modules and they dont all show up in the "change on every puppet run" makes me think that [16:32:57] there actually isn't, the flapping is caused by rsync not being configured and puppet trying to start it anyway [16:33:19] (there isn't 2 modules, from what I can see) [16:34:05] I was just speculating about the reason for the include being done in this way, which means there's no way to pass params to it [16:37:14] claime: I see. yea, we should be able to just replace the include with Deprecated JCraft JSch client library is replaced with MINA SSHD client library per default. There is still option to switch to using JCraft JSch client library. Support for JCraft JSch will be removed in the next gerrit release. [16:37:19] oops, bad paste:) [16:37:29] I meant to say, it can be: class { 'rsync::server': [16:37:35] instead [16:37:49] you can already see it being used like that in other places [16:37:57] modules/profile/manifests/doc.pp: class { '::rsync::server': } [16:38:33] and then there is a way to say "only if not already declared" [16:38:57] That will work if you only have one module [16:39:10] Resource like declaration will break on multiple [16:39:33] ensure_resources`. This function takes a list of resources and creates them if they do not exist. [16:40:05] I don't see a way to know that the rsync::server::module class is only used once [16:43:23] <_joe_> Krinkle: <3 [16:43:34] <_joe_> effie: we've finally been freed from session redis for good [16:43:49] I can live with that [16:43:53] <3 [16:44:16] Krinkle: next time we meet, first round is on me [16:44:24] akosiaris: I can expense that right? ^ [16:45:12] regardless, πŸ₯³πŸ’ƒπŸΌπŸ•ΊπŸ½ [16:46:08] :) [20:27:23] https://tech.trivago.com/post/2022-08-23-how-we-scaled-our-prometheus-setup/