[08:58:00] godog: i just noticed my pontoon env is failing on puppet: https://phabricator.wikimedia.org/P17838 [08:58:03] have you seen that before? [09:00:14] kormat: I have not seen that yet, I'm checking the o11y stack [09:03:59] kormat: yeah seeing the same with a recently rebased branch [09:04:56] I'm assuming that's due to the recent certs work, not sure yet what's the right fix [09:06:39] checking [09:07:53] elukey: thanks [09:08:32] my hunch is that those files are shipped by wmf-certificates debian package and that is expected to be installed on the puppet master ? [09:08:38] things are still in flux on that side, we are working on https://phabricator.wikimedia.org/T296089 [09:09:29] so on all nodes that run profile::base::certificates we are now generating a bundle .pem composed by the two certs brought it by wmf-certificates [09:10:09] (wmf-certificates now ships with its own way of creating one, so we are trying to merge the two approaches) [09:10:22] mm. wmf-certificates isn't installed on my pontoon env [09:11:03] and this is due to a recent change that I merged, it is installed only in the prod realm [09:11:15] elukey: pontoon may be missing a hiera config, lemme check [09:12:41] I was mistaken btw, the files in /usr/share/ca-certificates/wikimedia/ do exist on the puppet master e.g. pontoon-puppet-01.monitoring.eqiad1.wikimedia.cloud [09:13:10] godog: not on puppetmaster.mariadb104-test.eqiad1.wikimedia.cloud [09:13:44] I still get the same error heh [09:14:57] or I'm misreading and the files are expected to exist locally on the host ? [09:15:46] in the cloud.yaml hiera config we have set, globally, that trusted_certs is empty, so this creation of the bundle shouldn't happen [09:15:58] we have created a config only for deployment-prep [09:16:47] but I am ignorant about pontoon and its hiera config [09:17:16] (in cloud.yaml we have profile::base::certificates::trusted_certs: [] for example) [09:17:29] not sure if we need the same in pontoon.yaml [09:18:41] godog: is pontoon pickup up prod's hiera config? [09:19:00] it is yeah, ideally with minimal overrides [09:19:50] in a pontoon env though the root of trust is different, e.g. the puppet ca [09:21:31] so yeah we wouldn't want the public certs shipped by wmf-certificates anyways [09:21:49] created https://gerrit.wikimedia.org/r/c/operations/puppet/+/741847/ [09:22:47] elukey: i'll apply it locally and test. [09:22:54] ack [09:23:30] in theory this should allow the creation of a bundle composed by the local puppet ca cert, without any extra PKI (that is not configured for pontoon IIUC) [09:23:38] we do something similar for deployment-prep [09:24:07] this may go away depending on what we decide in https://phabricator.wikimedia.org/T296089 [09:24:21] yeah I was looking and in hieradata/cloud/eqiad1/deployment-prep/common.yaml we're using /var/lib/puppet/ssl/certs/ca.pem [09:24:43] ah yes it should be the same file in theory, I can change the code review [09:24:50] which AFAICT should be the same as /usr/local/share/ca-certificates/Puppet_Internal_CA.crt [09:25:02] yes yes [09:25:04] yeah, not sure which is the canonical location though, probably what you wrote? [09:25:28] good question, not completely sure [09:25:46] (brb) [09:26:04] elukey: works [09:27:02] kormat: from how you wrote it, I feel that you were very surprised about it [09:27:13] 👍 [09:27:28] lol yeah I deserve it I know [09:29:10] godog, kormat - I'll switch to /var/lib/puppet/ssl/certs/ca.pem for consistency with deployment-prep (John reviewed that bit at the time), and merge if you are ok [09:29:23] (the sha256sum of the files is the same) [09:32:42] maybe I can change the defaults [09:33:09] elukey: well, it's not like you can make it any worse! ;) [09:33:55] kormat: thanks for the constant support, it is really important for me [09:34:14] merged, [09:34:20] :D 💜 [09:34:45] the defaults of the profile can be improved, I'll review those in a bit [09:42:51] elukey: very nice, thank you ! [09:45:05] in a related note, a little while back I started with pki in pontoon since we'll need it, LMK if you (collective) would like to help [09:46:28] definitely yes [09:47:09] godog: o/ yes please [09:47:45] all the efforts for profile::base::certificates were done to have a "shared" way between cloud/deployment-prep/prod/etc.. to create bundles (p12,pem,jks) of the WMF internal root CAs (until a bright day in which we'll have only PKI) [09:48:45] the wmf-certificates way is also good, especially for Kubernetes-land [09:49:11] nice, thanks btullis elukey I'll keep you posted [09:50:33] I also had the same error in my fledgeling pontoon stack as kormat: but hadn't got around to asking about it. Now fixed with a rebase. Yhanks all. [09:50:42] ...Thanks... [09:57:45] btullis: is pontoon-netmon-01.monitoring.eqiad1.wikimedia.cloud yours, by any chance? [09:58:49] kormat: Not guilty :-) Mine is puppetmaster-01.data-engineering.eqiad1.wikimedia.cloud [09:59:00] damn, ok [10:00:50] oh, i should have paid closer attention. hi, godog ;) [10:01:11] dammit, I thought I made it [10:01:19] :D [10:01:29] godog: it's been sending cronspam [10:01:54] it's trying to reach `rancid-admin-core@monitoring.eqiad1.wikimedia.cloud`, and the bounces go to root@wm.o [10:03:12] mmhh ok, I'm taking a look [10:03:54] do you have a message-id or text I can search for ? I'm failing to find examples [10:05:34] godog: https://phabricator.wikimedia.org/P17839 [10:07:06] thank you [10:07:26] it all gets flagged as spam by gmail for me [10:08:49] I'd agree with that assessment [10:09:44] :) [11:23:27] https://twitter.com/bradfitz/status/1463559209740500994 [11:55:21] very cool :) [12:43:31] indeed! [12:54:26] jbond: QTE said they'd take ownership of deployment-prep but have never done so. Currently no one does until it's down and it's whoever with access & knowledge finds it first. [12:56:24] RhinosF1: thanks that was the impression i had [13:00:24] Cool, I saw your comment on a task where you said not sure. Hope I clarified / confirmed. [13:00:54] yep thanks [14:41:48] godog: good news! as of ~3h ago, the rancid cronspam is no longer showing up as a bounce. "yay". https://phabricator.wikimedia.org/P17839#91406 is an example. [14:42:25] "rancid cronspam" sounds like /r/rareinsults material [14:42:33] * kormat giggles [14:51:37] kormat: https://media.tenor.com/images/7d98d046b7464373f96c4e030b4ee8f6/tenor.gif [14:51:53] lol [14:52:10] kormat: I messed with rancid a little to get it to DTRT and eventually produced https://gerrit.wikimedia.org/r/c/operations/puppet/+/741919 [14:52:22] not what I was going for but it'll work [15:00:12] godog: nice work! [15:01:01] topranks: thank you sir [15:01:10] feel free to review too [15:01:20] godog: i saw the CR, it looks reasonable, but i'm going to leave it to the inestimable XioNoX to review [15:01:46] it's like you think I know rancid better [15:02:03] XioNoX: it's like i think you _deserve_ to know it better. ;) [15:02:12] it was there before I joined, and will be there after the end of the universe, it just runs [15:02:17] haha [15:02:22] lolz [15:02:28] godog: have you thought of disabling smtp completely in pontoon? [15:02:38] I'll have a look at the CR later today unless it's urgent [15:03:01] XioNoX: no not urgent, already applied in pontoon [15:03:47] kormat: only in passing really, like keep the daemon but blackhole (or log) everything ? [15:04:01] yeah something like that [15:04:18] it seems like the sort of thing that shouldn't be enabled by default [15:24:00] fair enough, probably that's the simplest solution [16:01:25] kormat: by way of counterargument, if you're using a pontoon stack for testing, having broken things that emit email not be a suprise when you push to production is probably useful? [though they should go to the stack owner, not root] [16:27:37] Emperor: then either you should be checking for failures, or very carefully enabling mails such that you don't spam root@ :P [16:53:29] I think there's a plan somewhere to have them go to project admins [16:53:32] It just needs work