[14:12:48] the moment yuo tell someone that in puppet `FooBar == "FOOBAR" == 'foobar'` https://media3.giphy.com/media/yITZV7ufSINtpEYbLr/giphy.gif [14:18:22] that's how I feel when someone tells me something about Puppet in general :P [14:21:18] lol :D [14:21:47] :ยด) [14:38:13] moritzm: https://www.irccloud.com/pastebin/UPircP23/ it seems like we have some issues with some apt keys [14:40:20] jbond: still not the worse: https://www.youtube.com/watch?v=D5xh0ZIEUOE (has been posted multiple times in the past) [14:40:45] dduvall: could you fix it please? [14:41:07] dduvall: it looks like you've added the terraform component for bullseye in 2384e363e58 [14:41:28] effie: ahh yes at least javascript is worse, is that the bar ;) :P [14:43:54] <_joe_> effie: also https://bash.toolforge.org/quip/AVfTAUmefIH_7EDsriqu [14:44:19] <_joe_> vgutierrez: he can hardly fix it given a) he's surely asleep b) doesn't have root [14:44:41] <_joe_> so someone else must have merged the change [14:45:01] <_joe_> mutante / moritzm ^^ [14:45:03] lol _joe_ [14:45:28] _joe_: so I guess I can revert it [14:45:37] <_joe_> vgutierrez: exactly [14:47:36] Daniel started to investigate the issue with the way that key gets integrated, but in fact let's revert for now [14:48:04] that package (to be used for some CI setup in cloud VPS) was imported using includedeb in the mean time [14:49:32] ack [14:50:33] revert at https://gerrit.wikimedia.org/r/c/operations/puppet/+/858315 [15:37:43] FYI, failing over the idp.w.o CNAME to idp1002 (which runs 6.6), should cause no impact [16:21:49] vgutierrez: yes, i added the component and there were issues after it was merged. mutante and moritzm were looking into it. see https://phabricator.wikimedia.org/T322344 [16:22:28] if you'd like me to submit a revert for the time being, i can, but i'm afraid that's the extent to which i know how to fix the issue [16:23:23] <_joe_> dduvall: I think valentin already did it :) [16:23:46] ok :) [16:24:38] this still warrants some investigation at some point, the failure was quite puzzling, but in the interim we're also fine with using includedeb as a workaround [16:25:06] it is a strange issue. it seems like the `InRelease` file signature is bad but not `Release.gpg`, so i suggested maybe adding `GetInRelease: no` but that's only based on my reading of the manpage. i have no real experience with this issue [16:26:01] and why does apt not have a problem with the repo but reprepro does? another strange aspect of it [16:38:00] is it me or helmfile !logs are duplicated? [16:38:21] eqiad + codfq, start + done [16:38:23] *codfw [16:38:27] ah!, sorry [16:41:06] robh, sukhe: https://phabricator.wikimedia.org/T323094#8403001 :( [16:45:38] XioNoX: we could have link flaps from anything. Just an ethtool command from puppet, or a random machine crashing or rebooting or imaging. [16:46:04] should we be worried about this everywhere? The potential for this issue seems rather dramatic [16:46:05] bblack: yep, that's the scary part [16:46:23] bblack: I updated the comment [16:47:19] yeah the VC model is basically-everywhere execpt eqiad-E/F and drmrs, right? [16:47:34] I don't think it's as clear cut as flapping==crash, but flapping *could* cause the crash, maybe other unknown conditions need to be met [16:48:08] bblack: yeah, different minor version though [16:49:05] ok [16:49:06] and esams is running junos 18 so most likely safe [16:49:37] yeah it could be eqsin specific due to models/revs, possibly. [16:50:07] was there any recent change to switch configs that might be related and helping to trigger? in which case eqsin might just be the first example we've seen but we could potentially trigger elsewhere. [16:50:39] nop, those has been quite stable/untouched for a while [16:52:29] I'm out next week, so I can look at doing the POPs upgrades when I'm back, should be a ~30min hard downtime per site [17:05:39] XioNoX: thanks [18:11:35] Apparently my calendar is wrong, it must be Apr 1 already again: https://datatracker.ietf.org/doc/rfc9078/ [18:53:20] I can confirm we had issues with the keys for the signed terraform repo and saw the backlog. reverting it was right. [19:31:23] cwhite: regarding the Metrics patches, was there a conversation somewhere that lead to those? Fine either way but just checking if there's context I can draw from in CR. [19:53:04] Krinkle: no conversation has been had yet. Expect one in the near future. Still putting a plan proposal together. [21:10:36] Hello, I'm working on an image with docker-pkg [21:10:49] The docs say [21:10:50] > Dockerfile.build.template (optional) a dockerfile template for a build container (see below) [21:10:51] https://github.com/wikimedia/operations-docker-images-docker-pkg/blob/b32ad662bbf60fac69e0bd5d1724169cc6219d01/README.rst#build-the-images [21:11:01] but there are no other references to a Dockerfile.build.template that i can find [21:11:05] is that a thing? [21:17:34] also, is there a way to use Docker build RUN caching while developing? i change one line at the bottom of my Dockerfile.template and it has to re-do the whole thing?