[13:03:02] I have a requestctl commit to review if anybody has time: https://requestctl.wikimedia.org/commit/b03c3c653e4a23782384f96623ab26e358ff7502 [13:06:54] elukey: looks good, committed [13:08:39] <3 [13:08:45] I am cleaning up some garbage :D [14:21:38] bjensen: can you catch me up on your memcached work? I'm seeing puppet breakage on cloudcontrols. [14:21:48] This is you adding ssl to memcached connections right? [14:22:06] hmmm, apologies, i must have misread output there [14:22:26] this is migrating memcacheds which use ssl to pki [14:22:39] i was under the impression that cloudcontrols did not use ssl, and would therefore not be impacted [14:23:27] this would have been the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1337942 [14:24:35] the pcc diff: https://puppet-compiler.wmflabs.org/output/1337942/7811/cloudcontrol1006.eqiad.wmnet/fulldiff.html [14:24:43] that's correct as far as I know, should just be local unencrypted connections [14:24:55] the error is just [14:24:56] Unknown variable: 'ssl_cert'. (file: /srv/puppet_code/environments/production/modules/profile/manifests/memcached/instance.pp, line: 193, column: 30) [14:25:12] so maybe just need to add a default or set a dummy in hiera [14:25:53] couldn't tell you why it passed pcc when it fails in reality. that's not supposed to happen! [14:26:05] oh, probably because of hiera diffs [14:27:30] andrewbogott: adding a default sounds right to me, i'm not quite sure the best way to go about that, though [14:28:05] ok, I'll dig a bit. It might not be a hiera thing actually... [14:28:46] the docs say that profile::memcached::instance takes an ssl_cert arg but it actually doesn't. So maybe that's something from your cleanup... [14:29:44] hmmm, right, it doesn't need to anymore [14:30:10] Lots of stuff in those comments that doesn't exist anymore [14:30:34] i'm wondering if the fact that hieradata/cloud.yaml still contains values for ssl_cert, ssl_key, and localcacert might be the problem? [14:31:36] those were removed from other hieradata roles in the cleanup [14:31:37] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1337942 removed the case where $ssl_cert/$ssl_key/$localcacert were set for non-tls users but they're still used in all cases as parameters to the memcached class [14:32:29] ah, right, okay [14:33:54] i'm not sure what the best way to fix that is [14:34:24] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338212 might help [14:34:45] andrewbogott: that'll break all the TLS using cases, no? [14:35:03] I thought bjensen's point was that there aren't any tls cases anymore [14:35:22] if there are then they must all be broken currently [14:35:24] we don't need to pass a cert location in the tls cases anymore [14:35:42] andrewbogott: I certainly see `profile::memcached::enable_tls: true` in a bunch of cases. why do you think they are broken now? [14:35:43] i believe the current set of broken cases are non-tls only [14:36:18] bjensen: the simplest fix would be to modify the current `if $enable_tls {}` statement to have an else block setting the relevant local variables as undef [14:36:22] oh, I see, taavi you're right [14:36:37] taavi: is that "~" in puppet-speak? [14:37:15] `~` is yaml speak, puppet just uses a plain `undef` for the equivalent [14:37:35] ahhhh okay [14:37:37] on it [14:40:50] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338215 [14:40:52] ? [14:42:18] looks ok, assuming pcc passes [14:44:10] pcc seems happy, and also thinks this is a no-op [14:45:45] alright, proceeding [14:45:46] yeah, and looking at the pcc logs the warnings are gone [14:46:21] taavi: _ah_, that is something i did not check, i will do so in the future [14:47:02] tbh that should be more than just a warning, i wonder if our prod puppet config and pcc are out of sync or something [14:47:14] andrewbogott, taavi: thanks very much for the help, and sorry for the noise [14:48:52] cloudcontrols look happy now. Thanks for the fix [14:49:13] 🙏 [20:39:01] herron, cjd91: unless there are objections, I'm going do some reimaging of sessionstore hosts. I expect no disruption in service, but...you know, famous last words. [20:39:28] No objections from me [20:39:35] Thank you for the heads up [20:40:20] urandom: ack ok [21:31:07] herron: do you know, off the top of your head, why debian-installer (with a reuse recipe) wouldn't have any of my volumes show up under /dev/mapper? [21:31:29] {pv,lv,vg}display all return no output [21:34:08] are the underlying sd devices present? [21:34:16] yes [21:40:21] anything interesting in say pvscan with verbose/debug output? [21:40:54] I'm not sure offhand, but the devices and partition table/type and headers all look good? [21:41:29] yeah, near as I can tell [21:42:26] I'm re-running the installer to see if it's repeatable (it's shocking to me how frequently it's not). [21:43:00] ok [21:45:36] it's repeatable [21:47:07] https://www.irccloud.com/pastebin/Li2jwTed/ [21:47:32] https://www.irccloud.com/pastebin/r4FbgrSN/ [21:47:36] weird [21:48:54] oh... it didn't assemble the raid [22:00:24] so if I had to guess, this machine had some disks added to it, and the superblocks probably weren't zeroed, and there is some array metadata from the previous machine [22:01:17] so d-i isn't assembling the raid, which is the pv, and so no pv, which of course means no vg or lv [22:07:34] makes sense [22:17:18] good gawd, did I actually salvage this dumpster fire? I think I did...