[08:40:25] jayme: mmhh currently that knob is not exposed in service::catalog no, probably the easiest "fix" is to turn the probe into a tcp one instead [08:41:19] godog: but there is a knob? :) [08:44:23] jayme: I misspoke, there is for prometheus::blackbox::check::http but not for service::catalog ATM [08:44:56] the knob being auto-generating the alert(s) for the probe from puppet, as opposed to a generic alert in alerts.git [08:45:34] setting auto renewal to say 10/12 days also would work of course [08:48:09] hmm.... [09:00:25] ah, the probe is using the LVS IP, not the endpoints. I was wondering why the expiry time is flapping :D [09:03:35] service catalog is indeed lvs based [09:04:12] yeah, makes sense mostly. Until it does not when the different hosts have different certs :) [09:05:26] heheh working as intended [09:10:48] ah...I now see it [09:11:09] it's not even about the short lifetime of the certs, those are valid for a month [09:14:41] but the PKI code does not refresh them early enough for the alert to not trigger [11:57:33] I'm currently doing a pass over hosts in insetup* roles to identify any Puppet5/Puppet7 gotchas (like insetup roles defaulting to Puppet 7 which then get a role applied which doesn't default to Puppet 5 yet, causing cert issues) [11:58:07] for o11y there's just one thing: [11:58:46] logging-hd2* are currently installed with the insetup::observability role which defaults to Puppet 7 [11:59:13] but we don't default to Puppet 7 in the general case yet and these will likely use a new role [12:00:04] so let's already go ahead and create a stub role for them (using the base profiles) and mark it as configured for Puppet 7 on the Hiera role level [12:00:20] then the later service build out will continue to use Puppet 7 [12:00:40] I can create a stub role, just tell me the name you prefer [12:57:43] moritzm: thank you, I'd say insetup::observability::puppet5 or _puppet5 I'm not attached [12:58:02] other than that I think we're fine to default insetup::observability to p7 [12:59:00] insetup::observability already uses Puppet 7 [12:59:20] indeed [12:59:36] this is done to make sure that when the actual rol eventually gets applied we don't run into any issues, IOW to ensure that the later role gets configured for P7 as well [13:01:05] yeah makes sense to me, thank you for reaching out! [13:01:11] can't wait for the migration to be over [13:05:56] It'll unfortunatetly drag on for a few more months, since we need to wait for all buster migrations to be completed [13:06:08] any preference for the name of the new role? [13:07:52] moritzm: insetup::observability::puppet5 SGTM [13:08:23] ack re: buster, we're making good progress on the alerting hosts and I'm expecting early next Q to start reimaging [13:09:57] no, you're missing my point: these hosts _are_ already using Puppet 7 by means of the insetup::observability role (which defaults to Puppet 7), we need to create a placeholder entry for the eventual logging-hd role to make sure they continue to use Puppet 7 when the actual service rampup happens [13:13:28] moritzm: my bad, thank you for the explanation, since it is a placeholder please go with logging::opensearch::data::hd [13:14:09] ack, thanks! I'll prepare something and will add you and Cole as reviewers [13:16:28] cheers [14:46:01] Hey kids! I'm trying to throw together optional puppet-free workflows on cloud-vps and I'm ready for someone besides me to test. Would it be useful for me to enable a puppetless debian image in the pontoon project? Or somewhere else? [14:48:09] andrewbogott: hey, yes please, the project is 'monitoring', thank you! I'm not sure when I'll be able to test though I'm subscribed to the "unmanaged instances" task [14:49:04] that's https://phabricator.wikimedia.org/T326818 [14:49:06] godog: bookworm ok? [14:49:13] andrewbogott: yeah totally [14:49:28] oh boy, you've gotten a lot of emails from that task lately :) [14:50:11] heheh indeed, thank you for working on that btw [14:50:31] gotta run one more test and then I'll ping you again [14:50:48] cheers [14:55:51] godog: coming back to the service::catalog probe thing from this morning: I could just not add the probes and instead include prometheus::blackbox::check::http to my profiles to more or less get the same, right? [14:56:19] jayme: yes that's correct [14:58:23] godog: ok, now you have a new available image in that project, 'debian-12.0-nopuppet'. You'll want to explore the 'Key Pair' tab for access; keys selected there will be added to the 'debian' user. [14:58:26] lmk how it goes! [14:58:55] andrewbogott: lovely, thank you very much [14:59:40] Hope it works! [15:00:49] godog: ok. I think I'll do that then [15:38:43] jayme: yeah that's fair, especially for kinda of a corner case wrt cert expiration [15:40:51] it also solves my problem of not checking the servers individually [15:41:09] indeed that too [15:41:14] https://gerrit.wikimedia.org/r/c/operations/puppet/+/982819 - if you have a minute [15:41:28] pcc is still running [15:45:18] jayme: as I'm reviewing the patch I realized this is more or less equivalent to "prometheus can / can't scrape the apiserver(s)" isn't it ? [15:45:35] hum [15:45:42] yeah...I suppose [15:46:01] is there a way to check if that fired in the past? [15:46:06] https://phabricator.wikimedia.org/T353233 [15:46:39] or maybe it's not...no [15:47:01] prometheus can't scrape for other reasons (like it's client-cert not being refreshed) [15:47:14] but that does not mean the apiserver is in trouble [15:48:04] that's true yeah, FWIW next Q we'll likely be upgrading prometheus so that problem should be fixed [15:48:16] anyways, patch LGTM [15:48:19] 🎉 [15:50:43] jayme: something else that occurred to me, you can customise things like runbook, dashboard, etc [15:50:50] all for later tho [15:51:11] I would have, if only I'd had a proper runbook 😇 [15:52:02] updated the patch setting severity to critical only for now, thanks! [15:52:45] neato, +1 [15:55:33] let's find out :-p [16:06:22] godog: ran puppet on a apiserver and prometheus1005 afterwards but I don't see a config change regarding that [16:08:06] oh and..does the resource name need to be unique in whole puppet? (as it is exported) [16:08:22] that is a good question, I don't know [16:08:54] eheh [16:09:06] re: prometheus1005 not updating, a bit of a shot in the dark but could you try again? [16:09:18] lol, sure :) [16:09:24] should I reboot it before? :p [16:09:36] hahah pls no [16:10:20] to explain: how much time did pass between the apiserver puppet run and the prometheus one? I don't know but wonder how atomic puppetdb updates are [16:11:45] jayme: godog: I suspect the k8s prometheus instance is missing the config to import blackbox stuff from puppetdb [16:12:19] taavi: ah that'd be much more reasonable than my guess [16:13:17] that is very plausible :) [16:13:41] which rises the question: should this be in ops? [16:13:52] I just put it into k8s prometheus instances because I can... [16:13:59] yeah prometheus::blackbox::import_checks is in ops and tools only [16:15:19] jayme: yeah ops SGTM [16:15:44] taavi: ack, I'll make a note to validate that via puppet types on 'prometheus_instance' [16:16:06] ack, changing to "ops" [16:18:46] jayme: feel free to self merge [16:19:08] wilco [16:24:37] jayme: I have to go shortly and will check back later too [16:25:42] ack. I'll leave a message on status [16:41:39] We're getting PuppetFailure alerts for elastic1107 , which is in insetup...is this expected? Don't think I've seen that before [16:42:58] godog: "works" now but it seems like all apiservers merged together module wise - which makes sense in a way. But with different parameters (certfificate_expiry_days) it probably depends on the order of resoureces as to which one "wins" [16:52:47] ah, not true - everything gets pretty mixed up aparently [16:53:06] target=https://[10.192.16.48]:6443/readyz msg="Error for HTTP request" err="Get \"https://10.192.16.48:6443/readyz\": x509: certificate is valid for kubemaster2002, kubemaster2002.codfw.wmnet, kubemaster.svc.codfw.wmnet, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster, kubernetes.default.svc.cluster.local, kube-apiserver, not ml-staging-ctrl2001.codfw.wmnet" [17:06:47] so basically "last defined resource wins" there as well (blackbox module that is) [17:08:11] def. good call on not making this paging initialls :D [17:08:16] *initially [17:31:06] all good now with https://gerrit.wikimedia.org/r/c/operations/puppet/+/982858 merged [17:33:25] apart fromt he fact that I messed up the certificate_expiry_days... 🤦 [17:45:10] https://gerrit.wikimedia.org/r/c/operations/puppet/+/982889 - not merging this today. Was messy enough already :) [17:47:33] I'm getting PuppetFailed alerts on a host that's inservice (elastic1107)...is this expected? [17:48:40] err...insetup that is [19:47:48] inflatador: puppetboard might have some more info about what's happening on that host: https://puppetboard.wikimedia.org/node/elastic1107.eqiad.wmnet [19:50:00] cwhite Gotcha. This is a new host that's still in with DC Ops and is insetup, so I wouldn't expect it to alert. There are a few other hosts (elastic1104-06) in the same state that don't seem to be alerting [19:50:23] seems some ssl issue `SSL_read: sslv3 alert certificate unknown` - I know there's some puppet upgrade work going on; might get more detailed info from Infrastructure Foundations team [19:50:33] Should I just set the host hiera variable that suppresses all alerts in the meantime? [19:54:07] If you think it's reasonable to suppress the alerts for elastic1107, I've no issue with that. [19:56:37] I'll suppress in AM for now, I might forget if I do it in puppet. Mainly I was curious about the insetup role and its effect on alerts. Prior to this, I had assumed alerts did not fire on insetup hosts [20:04:48] I don't see anything on the Alertmanager WT page about role::insetup and silenced alerts. It seems notifications are disabled automatically in icinga, though. [20:06:32] Ah, thanks for clearing that up. Sorry for not phrasing my question clearly [20:13:14] No problem, it's unclear to me as well if that behavior was replicated to alertmanager. I'd suspect not, but maybe g.odog will be able to confirm for us later. :)