[11:32:04] !log admin [codfw1dev] created project cloudvirt-canary [11:32:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:40:51] I’m filling out the cloud services survey. Are you interested in tools I develop outside of cloud services, or just the ones on cloud services? [18:43:16] harej: I suppose it depends on the particular question you are answering. If it's a question about what support you need to run the tool then I think you could answer with knowledge from any tools. If it's about uptime, support, etc then only tell us how much more awesome it would be if you ran it here. ;) [22:41:39] !log accidentally deleted deployment-sessionstore04 [22:41:40] urandom: Unknown project "accidentally" [22:41:58] !log deployment-prep accidentally deleted deployment-sessionstore04 [22:42:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [22:45:58] urandom: how is puppet breaking? If you are reusing the same host name there may be old puppet certs that need to be deleted. [22:46:26] bd808: ok, I though the same and so tried created a new one and got the same result [22:46:27] https://phabricator.wikimedia.org/P40867 [22:47:20] of course, ideally I'd recreate it with the same name so a mediawiki deploy wouldn't be necessary [22:47:57] urandom: that error message about the self-signed cert is mentioned in https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Step_2:_Setup_a_puppet_client [22:49:44] the puppetmaster for deployment-prep is supposed to be configured to autosign certs, but I think you still have to do the manual steps to drop the puppetmaster cert that the instance is born with. [22:50:23] there have been multiple attempts to automate that away but as far as I know none of them have worked [23:16:38] bd808: yeah, that's not working: https://phabricator.wikimedia.org/P40873 [23:18:04] ugh... you are actually getting closer. Now it's mad about the hostnames in the puppetmaster's signing cert. [23:18:37] oddly enough, what I currently have in Hiera Config for that vm is `puppetmaster: deployment-puppetmaster04.deployment-prep.eqiad.wmflabs`, but I did have deployment-puppetmaster04.deployment-prep.wikimedia.cloud at one point [23:18:58] (I changed it, and re-ran everything after it complained thusly the first time) [23:19:37] ah. once that hostname gets into /etc/puppet/puppet.conf (I think that's the file) puppet won't fix it up. You should be able to manually set it to the "real" value. [23:22:23] bd808: oh! yeah, that seems to have worked! [23:22:37] let me see if I can't get to this state with the (re)created vm [23:23:26] Hand editing things that Puppet should be managing is the digital version of percussive maintenance ;) [23:23:54] oh great, it's config was already correct...so it must have a different issue [23:24:18] it might just be `sudo rm -rf /var/lib/puppet/ssl` [23:25:11] but you may have to go deeper and do some cert management on the puppetmaster too if things are borked hard [23:26:52] yeah, it was both...seems OK now. So let me see if I can add the service config back in (also hiera, so 🤞) [23:49:11] bd808: I think it works...thanks! [23:50:45] urandom: \o/ I can login to meta.wikimedia.beta.wmflabs.org again. If I had an "I broke betawiki... and then I fixed it." sticker I'd send it down to you. ;) [23:50:58] haha