[02:13:52] <ori>	 I was going to merge James_F's beta-only https://gerrit.wikimedia.org/r/c/operations/puppet/+/980966 on puppetmaster (we're both at a Wikimedia NYC hackathon event). It's tested and safe. But there's another change in the diff: https://gerrit.wikimedia.org/r/c/operations/puppet/+/980950
[02:14:30] <ori>	 I can't vouch for the second one, so I'm leaving both unmerged. cc mutante ^
[05:20:23] <ori>	 andrewbogott: thanks for merging
[10:15:19] <hnowlan>	 ebernhardson: yep, sure 
[10:23:06] <moritzm>	 FYI; I'll switch cumin1001 to Puppet 7 in 10 minutes (cumin2002 has already been moved to Puppet 7 three weeks ago)
[10:58:21] <marostegui>	 moritzm: we need to revert that change
[10:58:34] <marostegui>	 we cannot connect to any db from cumin now
[10:58:57] <marostegui>	 # db-mysql db2110
[10:58:58] <marostegui>	 ERROR 2026 (HY000): SSL connection error: self signed certificate in certificate chain
[10:59:21] <moritzm>	 uh? 
[10:59:24] <moritzm>	 let me revert
[10:59:29] <marostegui>	 yes please :)
[10:59:41] <marostegui>	 we'd need to investigate what's the fix for that
[10:59:44] <moritzm>	 didn't anyone use cumin2002 for DB things in the last three weeks?
[10:59:53] <moritzm>	 we can use cumin2002 for that investigation :-)
[10:59:56] <marostegui>	 moritzm: I personally don't use cumin2002 nop
[11:00:25] <marostegui>	 but yeah, we need to investigate the fix for that
[11:14:59] <marostegui>	 I have created https://phabricator.wikimedia.org/T352974 to track the issue
[11:17:07] <moritzm>	 marostegui: can you retry from cumin1001?
[11:23:02] <moritzm>	 I've added T352974 as a blocker in https://phabricator.wikimedia.org/T349619
[11:23:03] <stashbot>	 T352974: puppet7 on cumin breaks database connections - https://phabricator.wikimedia.org/T352974
[11:24:47] <marostegui>	 works yep
[11:25:30] <marostegui>	 moritzm: it also works with the hosts we already have migrated to puppet7
[11:25:49] <marostegui>	 moritzm: arnaudb is going to investigate
[11:25:55] <marostegui>	 coordinate with him please
[11:26:05] <moritzm>	 I will
[11:26:29] <marostegui>	 thanks!
[12:09:12] <marostegui>	 moritzm arnaudb I added some stuff to the task after taking a quick look
[12:22:43] <arnaudb>	 ack, will investigate during afternoon
[13:32:01] <Amir1>	 puppet ci is running flake8 on python2 and it's giving me syntax error. Can we just bump it to three?
[13:32:04] <Amir1>	 https://www.irccloud.com/pastebin/qNmEPT9c/
[13:39:03] <Amir1>	  https://gerrit.wikimedia.org/r/981329
[13:40:15] <volans>	 it runs flake8 on py2 for py2 files and py3 for py3 files IIRC
[13:40:27] <Amir1>	 but it doesn't :D
[13:42:01] <volans>	 because it has no shebang
[13:42:09] <volans>	 see sort_python_files in rake_modules/taskgen.rb
[13:42:15] <Amir1>	 aah
[13:42:22] <volans>	 we can probably invert the logic at this point
[13:42:28] <volans>	 but we do still have py2 stuff unfortunately
[13:43:05] <volans>	 and do py3 by default and fix the py2 cases if needed
[13:43:46] <Amir1>	 yeah
[13:43:49] <Amir1>	 sgtm
[13:44:11] <volans>	 could you open a task with puppet-core tag?
[13:44:40] <volans>	 sorry, "puppet CI" one, seems more appropriate
[13:44:49] <Amir1>	 sure
[13:45:22] <volans>	 thx
[15:00:23] <inflatador>	 do we have puppet code to enable detailed envoy logging? I'm looking at /var/log/envoy on my wdqs host and there ain't much there ;(
[15:01:21] <_joe_>	 inflatador: not puppet code, you can use curl
[15:01:37] <_joe_>	 but yes also puppet code
[15:02:44] <_joe_>	 ah wait
[15:02:52] <_joe_>	 you want /access logs/ to your local service
[15:02:59] <_joe_>	 because the backend doesn't have it?
[15:03:04] <inflatador>	 _joe_ based on https://wikitech.wikimedia.org/wiki/Envoy#Runtime_configuration , looks like I have to turn it on thru cURL first?
[15:03:36] <inflatador>	 I'm having issues where a request never makes it to nginx...envoy is in front so I'm trying to figure out why
[15:04:28] <inflatador>	 seems to only happen when I use a query string if that helps
[15:04:54] <_joe_>	 inflatador: oh ok, interesting
[15:05:09] <_joe_>	 inflatador: so yeah, our local tls terminator has no logging atm 
[15:05:54] <inflatador>	 _joe_ that's OK, just wanted to make sure before I make any changes
[15:05:57] <_joe_>	 you can manually disable puppet and add the stanza for logging to the local service entry
[15:06:20] <_joe_>	 send a patch my way, but I fear I won't be able to take a look until monday
[15:06:32] <inflatador>	 no worries...I'll one-off it. Thanks for your help
[15:06:37] <_joe_>	 but r.zl and j.ayme and h.nowlan should also be able to help
[18:41:22] <inflatador>	 Has anyone gotten puppet failures from the using ipresolve function with hostnames and AAAA records? error is https://puppet-compiler.wmflabs.org/output/980499/270/wdqs1015.eqiad.wmnet/change.wdqs1015.eqiad.wmnet.err and code is at https://gerrit.wikimedia.org/r/c/operations/puppet/+/980499/13/modules/profile/manifests/query_service/monitor/ldf.pp#21 
[18:49:51] <sukhe>	 inflatador: not that I know everything that is happening here but by use ipresolve on the fqdn and not ip or something directly?
[18:50:03] <sukhe>	 $facts['networking']['ip'] and ip6 I mean
[18:50:15] <sukhe>	 or even ipaddress and ipaddress6 
[19:07:43] <inflatador>	 sukhe good point, let me try that