[07:44:38] <elukey>	 vgutierrez: o/ around? update-ocsp-all is logging some exceptions in the logs, some cp nodes started to emit OCSP freshness warnings/criticals
[07:45:25] <elukey>	 seems mostly for digicert
[07:50:05] <elukey>	 I see the issue is https://gerrit.wikimedia.org/r/c/operations/puppet/+/854608
[07:50:09] <elukey>	 Cc: sukhe: --^
[07:53:46] <elukey>	 added a note to https://phabricator.wikimedia.org/T321309
[07:56:32] <elukey>	 check_output_errtext calls popen, probably it returns a byte array that needs to be converted to str
[07:56:35] <elukey>	 or similar
[07:59:23] <elukey>	 yep seems that communicate() returns a bytearray
[08:06:54] <elukey>	 created https://gerrit.wikimedia.org/r/c/operations/puppet/+/856126, in theory it should DTRT but I am very ignorant about the ocsp stack so I'll let you folks -2 it and use something else in case :)
[08:11:32] <elukey>	 or we can simply add text=True to open
[08:15:20] <elukey>	 (updated)
[08:22:39] <vgutierrez>	 Hmm
[08:24:40] <vgutierrez>	 thanks for submitting the CR
[08:25:12] <vgutierrez>	 I'm not 100% sure that's the root cause here though
[08:25:40] <vgutierrez>	 I'm seeing digicert-2021 material still deployed
[08:25:51] <vgutierrez>	 and that of course is gonna fail cause that cert is already expired
[08:26:43] <vgutierrez>	 indeed
[08:27:25] <vgutierrez>	 let me fix that
[08:32:31] <elukey>	 yep yep I think there is a problem with the script and one that is related to the cert, but the python code shouldn't fail in that way imho
[08:33:47] <vgutierrez>	 yep.. so both things are required
[08:34:12] <elukey>	 I know that you were waiting for me to enlight your sunday
[08:34:21] <vgutierrez>	 of course :)
[08:34:23] <elukey>	 :D
[08:35:10] <vgutierrez>	 I've manually tested your change in cp1075
[08:35:34] <vgutierrez>	 looks good
[08:36:33] <elukey>	 text=True was added IIUC from py3.7+, but it should be fine for us 
[08:46:13] <vgutierrez>	 cleaning up digicert-2021 related OCSP material
[08:46:13] <elukey>	 vgutierrez: can i help in any way? (run puppet, check the scritp, etc..)
[08:46:25] <vgutierrez>	 cumin is doing the heavy lifting right now
[08:46:36] <elukey>	 super
[08:59:53] <elukey>	 I see recoveries on the nodes (the script doesn't fail anymore) but icinga seems slower to recover
[09:00:05] <elukey>	 I can force  the rechecks
[09:00:33] <elukey>	 (lemme know when cumin is finished)
[09:01:09] <vgutierrez>	 first stage is cleaning the digicert-2021 stuff and triggering puppet runs
[09:01:34] <vgutierrez>	 now we need to refresh the OCSP stapling data for the 2022 one
[09:04:24] <vgutierrez>	 ocsp refresh is ongoing.. is gonna take a little bit (-b2 -s30)
[09:05:03] <elukey>	 ah ok, I checked on 1077 and I saw "update-ocsp-all.service: Succeeded." so I thought it ran right after puppet changed the .py
[09:05:32] <elukey>	 recoveries are coming, nice :)
[09:05:50] <vgutierrez>	 sadly puppet doesn't cover cleaning up ocsp-update config for removed TLS material
[09:06:08] <vgutierrez>	 nor cleaning /var/cache/ocsp
[09:17:49] <elukey>	 vgutierrez: thanks a lot! 
[09:28:45] <vgutierrez>	 no problem
[09:28:53] <vgutierrez>	 it looks like I'm gonna be able to get my breakfast now :)
[09:29:31] <vgutierrez>	 enjoy your Sunday elukey <3
[11:59:29] <sukhe>	 thanks all! <3 just caught up
[18:04:00] <vgutierrez>	 sukhe: np