[07:55:03] gehel@cumin1004:~$ sudo cumin "A:trixie" [07:55:03] 1 hosts will be targeted: [07:55:21] Some how, I doubt that we've migrated everything to Forky... [07:55:38] Anyone knows if something happened to puppet / facter? [08:04:13] lovely! [08:05:35] you can use facts directly, I'm not sure what this trixie alias is about [08:05:35] sudo cumin 'P{I:facts.os.distro.codename = trixie}' returns 1k+ hosts [08:05:53] IIRC A:trixie should work though [08:05:56] lemme check [08:06:22] A:trixie is an alias for `trixie: P{F:lsbdistcodename = trixie}` [08:06:36] yeah [08:07:02] could it be https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689 ? [08:07:38] probably not mmm [08:07:54] seems like lsb_release has been deprecated... [08:08:59] patch coming up [08:09:12] jelto's way is most likely the cleanest way going forward [08:09:24] (more standard, etc) [08:09:33] I just copied it from https://wikitech.wikimedia.org/wiki/Cumin :D [08:09:47] I am wondering if it is related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347758 [08:11:06] elukey: that seems like a good candidate yeah [08:11:29] and also https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689/3/modules/puppetdb/functions/query_facts.pp [08:11:52] because cumin behind the scenes calls the puppetdb api [08:12:21] rollback or roll forward? [08:12:25] maybe it is just the lsbdistcodename format that we have in the cumin aliases that needs to be fixed [08:12:34] If someone has a minute to review : https://gerrit.wikimedia.org/r/c/operations/puppet/+/1348433 [08:14:16] that lgtm but I'll leave to to more experts eyes [08:15:09] gehel: how did you test it? [08:21:12] Jelto's version seems working, I can't make your working (I think it needs a replacement s/F/I) [08:21:20] *yours [08:22:00] elukey: they both work [08:22:30] P{F:os.distro.codename = trixie} [08:22:30] P{I:facts.os.distro.codename = trixie} [08:23:08] oh right my bad [08:23:19] I made the same mistake when testing them :) [08:23:43] I tested:sudo cumin "P{F:os.distro.codename = trixie}" [08:23:43] 1172 hosts will be targeted: [08:23:54] yep yep sorry my bad [08:24:25] because I am reasonably sure it is related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689/3/modules/puppetdb/functions/query_facts.pp, and the os distro aliases are the only one using F: [08:24:38] (there is also the puppet7 one but we don't really use it) [08:24:53] the puppet7 alias still works [08:26:09] puppetversion was/is no legacy fact, so no impact expected there [08:26:27] gehel: the patch LGTM [08:27:05] moritzm: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1348433 looks sensible, wdyt? I don't see anything else in the alias erb that could be potentially impacted [08:27:18] but I am a little ignorant in spotting legacy vs structured facts to be honest [08:27:24] elukey: already +1d 30 seconds ago :-) [08:28:24] os-reports seems also impacted, I'll make a patch for it later (but little practical impact failing, just internal to SRE) [08:28:25] merged [08:29:07] thanks gehel! [08:29:08] thanks for the help [08:33:25] and I can confirm the cumin alias now works [08:43:33] bjensen, XioNoX: hello oncall there is a high packet loss since 6:20 UTC on one of the dallas-singapore circuits. I'm going to drain the problematic link. [08:44:39] ack, thanks! [09:00:08] tracked in https://phabricator.wikimedia.org/T439947 [13:25:05] gehel: sorry about the breakage on facts, I should have sent a note out last night after leaving the flag flipped, I'll do that now [13:31:55] jhathaway: no problem. It was a good occasion to update those deprecated facts... [13:47:11] * Lucas_WMDE continues being tempted to !bash all those quotes about updating deprecated facts [13:56:47] Lucas_WMDE: make sure you are using the latest version of the official SRE-approved facts [13:57:17] I’m going to make my own facts, with— [13:58:30] :D a missed opportunity in my email! [14:59:53] https://gitlab.wikimedia.org/lizf/dotfiles/-/blob/bobby/.spacemacs?ref_type=heads#L604-619 any other emacs users here? I made a gerrit generator for git-link [17:12:20] https://www.phoronix.com/news/Linux-Patches-Toward-DRBD-9 looks like the kernel DRBD is on track to get a major update...wonder if that will help our Ganeti clusters at some point [17:35:14] hmm. RuntimeError: Mismatched FQDN zuul1001.eqiad.wmmet does not match Netbox FQDN zuul1001.eqiad.wmnet [17:35:25] looks like matching though? [17:36:07] trying to use decom cookbook but it was interrupted last run, now it's in zombie state. so I ran it again with --force and this happens [17:36:38] the first one is 'wm_m_et' for whatever reason [17:36:38] question is how to properly decom it now. I still see it in netbox. [17:37:06] taavi: aww:) and that's why I needed a second pair of eyes, clearly. ty [18:27:15] today, when someone has trouble accessing some of our resources due to an IP block, they get a screen in english (Requests from your IP have been blocked, etc etc ...). is there a way for this to be presented in whatever language a person operates in? (or for the screen to have more than just english)