[16:16:20] <jbond>	 hi all i noticed the same error on at least two projects that hav there own puppetdb instance (sso, toolsbeta) the issues was that /etc/ssl/certs/wmf-ca-certificates.crt was empty (im not sure why)
[16:16:29] <jbond>	 the fix was to to the folllowing
[16:16:37] <jbond>	 sudo cp /var/lib/puppet/ssl/certs/ca.pem /etc/ssl/certs/wmf-ca-certificates.crt 
[16:16:48] <jbond>	 sudo systemctl stop puppetdb
[16:17:04] <jbond>	 sudo systemctl restart postgress@11-main.service
[16:17:18] <jbond>	 sudo systemctl start puppetdb
[16:43:15] <andrewbogott>	 thanks jbond!  That was tools and deployment-prep?
[16:43:28] <andrewbogott>	 That's likely the only places running puppetdb as far as I know but I can check
[16:44:23] <jbond>	 andrewbogott: i fixed sso and toolsbeta i didn;t check any where elses but i can check deployment prep, one sec
[16:44:41] <jbond>	 if you can let me know the fqdn of the db on tools i can look at that as well
[16:44:50] <andrewbogott>	 welp, apparently there are more puppetdbs than I know
[16:44:53] <andrewbogott>	 sure, one second
[16:45:28] <andrewbogott>	 	tools-puppetdb-1.tools.eqiad1.wikimedia.cloud
[16:51:49] <jbond>	 andrewbogott: tools and deployment-prep also fixed now
[16:51:59] <andrewbogott>	 thank you!
[16:53:02] <jbond>	 im 99% sure this is something i did as i have deinetly been playing with ssl recently but im not sure.  a) what the root cause of causing that file to be empty b) why it seemd to start ~9 hours ago :/
[16:53:11] <jbond>	 ill explore more on monday
[16:55:19] <andrewbogott>	 I'm trying to scan for other uses of puppetdb but it's taking forever
[16:57:04] <andrewbogott>	 bah, cumin is degraded somehow. Too deep of a rabbithole for Saturday