[07:26:03] good morning! [07:26:20] afaics the kafka keystore dinamic reload doesn't work :( [07:27:43] https://issues.apache.org/jira/browse/KAFKA-7429 is also not great.. [07:27:50] (we have 1.1) [08:44:29] <_joe_> elukey: yeah we'd have to reengineer the thing [09:06:40] _joe_ my idea for the moment is to set the default expiry for kafka tls cert to a long period, like 1y or similar [09:06:50] to avoid frequent rolling restarts [09:07:10] after/if we migrate to kafka 2.x we'll see [10:11:13] volans: i can't see where it's specified how spicerack uses modules in a cookbook's dir [10:11:36] * volans can't parse the question [10:11:39] i'm guessing i can just nuke an individual module and i don't need to update anything? [10:11:45] what do you mean? [10:11:50] `git rm cookbooks/sre/switchdc/mediawiki/09-update-tendril.py` [10:12:02] is that safe? [10:12:10] totally [10:12:17] i can't see any docs on whatever magic is used to find/use modules [10:12:55] doh, thanks, let me add some lines if I can't find them either [10:13:13] 👍 [10:13:25] https://doc.wikimedia.org/spicerack/master/introduction.html#cookbooks-hierarchy is what i was looking at, primarily [10:14:11] yep [10:14:26] anyway it's dynamically traversed when you run 'cookbook' the binary [10:14:56] volans: in alphabetical(ish) order? [10:16:33] yep, what python's pathlib's glob returns [10:16:36] more or less [10:16:53] now, in your specific case (and I might need to add something smart to puppet) [10:17:16] we'll need to remove the cached compiled file too, but I can take care of it and then look if there is a nice way to add it to puppet [10:19:31] oh, lovely [10:19:36] volans: is there CI for the cookbook repo? [10:20:23] or do i +2 for C and V? [10:20:52] there is CI [10:21:03] just C+2 when jenkins is happy [10:21:14] so for this specifically we can run: [10:21:17] sudo cumin 'cumin1001*,cumin2002*' 'rm -v /srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/__pycache__/09-update-tendril.cpython-3*.pyc' [10:23:47] I mean, the cookbook will disappear anyway because it will collect only the real files and ignore __pycache__ dirs, but to prevent weird behaviours in case of cross imports it's better to keep it clean [10:24:20] at the same time is probably better to avoid deleting cached files at each puppet run as they might cause race conditions with running cookbooks (I think, not 100% sure) [10:24:32] 😬 [10:27:25] volans: deployed, and pyc files cleaned up [10:29:00] \o/ [11:41:04] hey volans: I had a reimage get interrupted and now I get an error because the dhcp overrides are still in place - can/should I edit /etc/dhcp/automation/proxies/ttyS1-115200.conf manually to fix this or is there a better route? [11:41:52] hnowlan: interrupted how? on error it should cleanup the temporary DHCP settings [11:42:11] if you tell me from which cumin hosts you run it and what was the target hostname I can check the logs [11:43:00] volans: connection dropped and the controlling terminal disappeared :/ so wouldn't be surprised if graceful cleanup wasn't possible. it was run from cumin1001, target hostname is restbase1024 [11:43:28] are you sure is not still running? reimages are forced to be run on tmux/screen so a connection drop should not affect it at all [11:44:17] * volans checking [11:48:10] It doesn't seem to be no [11:49:07] ack, then feel free to rm /etc/dhcp/automation/ttyS1-115200/restbase1024.conf on install1003 and re-run the cookbook [11:50:39] volans: cool, thanks! [11:54:35] I have updated https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate after the recent findings about the dynamic reload feature [11:55:03] and also extended the default tls cert validity window to 1y for the kafka pki intermediate [12:40:06] we have less than 100 tasks left till we reach T300000 [12:43:40] :D [12:45:22] If I keep filing dupe tasks about my own tasks we'll soon reach that number marostegui :P [13:02:27] hello and happy monday [13:02:53] may someone puppet-merge a change for me please? It is for Gerrit and it is actually a noop for production https://gerrit.wikimedia.org/r/c/operations/puppet/+/756111 [13:03:29] it renames a file, a change that already got applied last year and got reverted in puppet but the file is still on the server and is the one being actively used now [13:03:41] so the change above merely catch up with today reality ;) [13:04:10] * jbond hashar looking [13:04:29] i gave way too many details in the commit message and as a reply for historical purpose :) [13:07:45] hashar: merged [13:07:54] jbond thank you very much [13:11:02] np [17:14:29] mutante: cool, miscweb! is the intention to have a single apache server per pod with virtualhosts serving different sites? [17:14:43] we have random apache stuff that would be nice to not have to run on ganeti or colocated [17:16:56] like: https://github.com/wikimedia/puppet/blob/production/modules/profile/templates/idp/client/httpd-yarn.erb [17:24:56] <_joe_> ottomata: that might need to be better revisited once we have ingress active [17:25:12] <_joe_> we might even rethink how we build miscweb at all [17:25:26] aye k [17:48:49] great presentation, Amir1 [17:49:40] Thanks ^^ [17:56:55] +1