[21:48:53] <bd808>	 andrewbogott: any idea what `Notice: /Stage[main]/Labs_lvm/Exec[create-volume-group]/returns: /usr/local/sbin/make-instance-vg-ephem: did not find applicable disk` might be about? -- https://phabricator.wikimedia.org/P74287
[21:49:48] <andrewbogott>	 any chance your volumes are vda/vdb and the puppet code is expecting sda/sdb?
[21:52:24] <bd808>	 `lsblk` shows sda and sdb with sdb1 (/), sdb14 (swap?), sdb15 (/boot/efi)
[21:53:11] <bd808>	 so the weird is that sda is the unpartitioned & mounted disk I guess?
[21:53:40] <andrewbogott>	 yeah... they're swapped from what puppet is expecting
[21:54:15] <andrewbogott>	 This is likely debian doing a coin-toss at boot time
[21:54:18] <bd808>	 what do I do about that? My disk level linux fu is very stale...
[21:54:20] <andrewbogott>	 but maybe you can fix it in fstab?
[21:54:41] <andrewbogott>	 This is a running gag with new linux distros, someone decided that predictable dev names was a bug
[21:55:00] <bd808>	 systemd jerks I would imagine
[21:55:12] <andrewbogott>	 lol fstab only uses the uuid and doesn't mention the device label
[21:55:14] <andrewbogott>	 so...
[21:55:15] <andrewbogott>	 hm
[21:55:48] <andrewbogott>	 I guess that puppet class should be refactored to take a uuid from hiera, or something messy like that.
[21:56:06] <andrewbogott>	 did all the new VMs show up with that same pattern or is it 50/50?
[21:57:09] <andrewbogott>	 https://www.reddit.com/r/linuxadmin/comments/nxfvjt/devsda_and_devsdb_keep_swapping/
[21:57:13] <bd808>	 this is the first one Tyler handed to me to figure out WTF. It had a puppet cert singing problem that I fixed by being global root
[21:57:31] <bd808>	 heh. I'm reading that same SO post
[21:57:49] <andrewbogott>	 "If MUST you can clamp/force certain devices to get a specific dev name by using udev" I'm not sure how to do that
[21:58:29] <andrewbogott>	 That top-rated answer reads to me like "computers are unreliable, never use them"
[21:58:45] <andrewbogott>	 You don't want to know the horror that our partman recipes have become in order to deal with this same issue
[22:00:14] <andrewbogott>	 bd808: in this case maybe you can just format that drive and update fstab by hand, and then puppet will skip that phase and get on with it?
[22:00:28] <bd808>	 I'm on https://unix.stackexchange.com/questions/362230/sda-and-sdb-keep-on-swapping and I guess if we could use `/dev/disk/by-uuid` labels then `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0` == expected sda and `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-1` == expected sdb
[22:01:14] <bd808>	 andrewbogott: maybe...
[22:01:49] <andrewbogott>	 if this only happens rarely you could also just throw that VM away and make a new one
[22:01:52] <andrewbogott>	 ^ what I would do
[22:03:50] <bd808>	 my brain does not love that solution yet, but I'll hang on to it
[22:04:14] <andrewbogott>	 ok.  I'm also willing to hack on that puppet code and see if it can be made predictable, but it's a bit late in the day to dive into that.
[22:04:32] <bd808>	 yeah, don't worry about it yet
[22:04:48] <bd808>	 I'll send up more flares if I stay stuck...
[22:05:27] <andrewbogott>	 I don't think this is a recent change (the unpredictable labels) so it seems unlikely that this has been happening that frequently
[22:05:34] <andrewbogott>	 since we have all those other working runners
[22:06:33] <bd808>	 well.. here is a fun thing: this doesn't actually fail the puppet run. It just ends up without the ephemeral disk
[22:07:08] <bd808>	 so I could be all over the cluster for all I know
[22:09:57] <andrewbogott>	 uhoh
[22:11:40] <bd808>	 ah ha! there is a way to twiddle the /dev/sda to /dev/sdb via hiera :)