[08:28:35] <elukey>	 hello folks
[08:29:41] <elukey>	 I don't see anything ongoing, I am  going to depool cp4037 to apply the changes to varnishkafka
[08:29:47] <elukey>	 hopefully this time it will work
[08:29:50] <elukey>	 (fingers crossed)
[08:42:14] <elukey>	 nope doesn't work
[08:42:15] <elukey>	 sigh
[08:48:23] <elukey>	 very interesting - if I comment 
[08:48:23] <elukey>	 #kafka.ssl.sigalgs.list=ECDSA+SHA256
[08:48:24] <elukey>	 it works
[08:53:00] <elukey>	 weird
[08:54:18] <elukey>	 jbond: o/ do you have a min for a brainbounce?
[08:57:02] <jbond>	 elukey: hi
[08:58:23] <elukey>	 jbond: morning :) I am seeing something weird in varnishkafka, namely that the ssl sigalgs setting causes the TLS handshake to fail (vk -> kafka broker)
[08:58:51] <elukey>	 do I need any extra setting for the vk's client cert? I am fairly ignorant in the TLS options for cfssl
[08:59:46] <jbond>	 eluke one sec let me check a few things
[08:59:54] <elukey>	 sure sure!
[08:59:57] <elukey>	 thanks :)
[09:00:24] <elukey>	 one thing that I had to fix manually is the cert's file, vk prefers the chained one afaics
[09:00:37] <elukey>	 (in puppet I've set the "cert" one not "chained")
[09:02:24] <jbond>	 elukey: i just checked and /etc/varnishkafka/ssl/kafka__varnishkafka_kafka_11.pem has `Signature Algorithm: ecdsa-with-SHA512` could you try setting afka.ssl.sigalgs.list=ECDSA+SHA256
[09:02:28] <jbond>	 afka.ssl.sigalgs.list=ECDSA+SHA256
[09:02:38] <jbond>	 third time lucky: kafka.ssl.sigalgs.list=ECDSA+SHA512
[09:02:45] <elukey>	 ahhhh
[09:07:16] <elukey>	 jbond: tried but I get again handshake failures sigh
[09:08:16] <jbond>	 elukey: im gussing when yuo leave it blank it falls back to some default list.  is there some way to see in the logs what it ends up using?
[09:09:13] <elukey>	 jbond: no idea, the logs are not that verbose
[09:09:29] * jbond just reading https://phabricator.wikimedia.org/T182993
[09:12:10] <elukey>	 I am trying to see if the kafka jumbo brokers are set to avoid the 512 sigalgs
[09:12:47] <jbond>	 ah hpossible, vgutierrez may be able to help here as well
[09:13:27] <vgutierrez>	 morning sir
[09:13:35] <elukey>	 aloha
[09:14:06] <vgutierrez>	 so... sigalgs set to ECDSA+SHA256 is not longer working?
[09:14:21] <jbond>	 correct
[09:14:37] <jbond>	 the new cert has `Algorithm: ecdsa-with-SHA512`
[09:14:49] <jbond>	 setting `Algorithm: ecdsa-with-SHA512`` also seems to fail
[09:14:54] <vgutierrez>	 lovely
[09:15:04] <jbond>	 setting kafka.ssl.sigalgs.list=ECDSA+SHA512 seems to fail
[09:15:05] <vgutierrez>	 could you point me to the endpoint using that cert?
[09:15:18] <vgutierrez>	 and I guess that's running TLSv1.3, right?
[09:15:25] <elukey>	 kafka-jumbo1001.eqiad.wmnet:9092 vgutierrez 
[09:15:30] <elukey>	 err sorry 9093
[09:15:32] <vgutierrez>	 :)
[09:15:35] <vgutierrez>	 thx
[09:15:40] <elukey>	 I see the following in the kafka broker's profile
[09:15:56] <elukey>	 # Note that this class configures java.security to set jdk.certpath.disabledAlgorithms
[09:15:59] <elukey>	 # to restrict the types of sigalgs used for authentication certificates via
[09:16:02] <elukey>	 # the profile::java hardened_tls parameter.
[09:16:53] * vgutierrez double-checking
[09:17:01] <vgutierrez>	 kafka-jumbo1001.eqiad.wmnet:9093
[09:17:03] <vgutierrez>	 right?
[09:17:09] <elukey>	 yep
[09:17:29] <vgutierrez>	 https://www.irccloud.com/pastebin/ZjeZ58qJ/
[09:17:43] <vgutierrez>	 ECDSA+SHA256 seems to be ok according to OpenSSL
[09:18:00] <vgutierrez>	 and BTW, it's using TLSv1.2, TLSv1.3 isn't available
[09:18:18] <elukey>	 yeah but I am not 100% sure if kafka supports it
[09:18:32] <vgutierrez>	 elukey: WDYM?
[09:18:58] <elukey>	 vgutierrez: https://issues.apache.org/jira/browse/KAFKA-7251
[09:19:04] <elukey>	 we still run java 8, not 11
[09:19:23] <elukey>	 plus the support seems "fixed" for kafka 2.5, and we have 1.1
[09:19:25] <vgutierrez>	 oh you were talking about TLSv1.3, ok
[09:19:36] <elukey>	 yes yes
[09:20:28] <moritzm>	 also, one thing to consider here: Kafka uses a security.java file with some stricter setting than the defaults
[09:20:54] <vgutierrez>	 elukey: which varnishkafka config do I need to check?
[09:21:01] <vgutierrez>	 elukey: /etc/varnishkakfa/webrequest.conf?
[09:21:16] <elukey>	 vgutierrez: correct yes
[09:21:37] <vgutierrez>	 elukey: I'm on cp4037 so this seems "dirty"
[09:21:53] <elukey>	 vgutierrez: ?
[09:22:02] <vgutierrez>	 #kafka.ssl.sigalgs.list=ECDSA+SHA512
[09:22:08] <vgutierrez>	 that comment I guess it's not on puppet
[09:22:16] <vgutierrez>	 any other changes I should be aware of? :)
[09:22:33] <elukey>	 nono it is mine, I also switched to the "chained" certificate file
[09:23:12] <vgutierrez>	 ack, give me one sec plz
[09:23:59] <elukey>	 moritzm: I tried to check /etc/java-8-openjdk/security/java.security but I don't see any explicit disable for some sigalgs afaics..
[09:27:31] <vgutierrez>	 elukey: the only change is the PKI migration, right?
[09:27:38] <elukey>	 correct
[09:29:20] <elukey>	 at this point it may be how I requested the cert? It contains CN:varnishkafka, that's the only special thing
[09:29:49] <moritzm>	 yeah, IIRC we're setting legacyAlgorithms for DSA, SHA1 (plus some settings related to Kerberos)
[09:30:15] <jbond>	 elukey: vgutierrez: its worth notoing if missed that with the line commented out, i.e. letting thing negotiate then it works
[09:30:25] <vgutierrez>	 err
[09:30:41] <jbond>	 elukey: thats correct right or did i miss interprit?
[09:30:41] <vgutierrez>	 so with openssl s_client and mtls 
[09:30:47] <vgutierrez>	 Peer signing digest: SHA256
[09:30:47] <vgutierrez>	 Peer signature type: ECDSA
[09:30:48] <elukey>	 jbond: correct
[09:30:58] <vgutierrez>	 so that's sudo openssl s_client -connect kafka-jumbo1001.eqiad.wmnet:9093 -tls1_2 -cipher ECDHE-ECDSA-AES256-GCM-SHA384 -cert /etc/varnishkafka/ssl/kafka__varnishkafka_kafka_11.chained.pem -key /etc/varnishkafka/ssl/kafka__varnishkafka_kafka_11-key.pem -chainCAfile /etc/ssl/certs/wmf-ca-certificates.crt
[09:32:25] <vgutierrez>	 appending -signalgs ECDSA+SHA256 also works
[09:32:31] <vgutierrez>	 SHA512 doesn't
[09:32:57] <vgutierrez>	 Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
[09:32:57] <vgutierrez>	 Shared Requested Signature Algorithms: ECDSA+SHA256
[09:33:05] <vgutierrez>	 ECDSA+SHA256 should be perfectly fine
[09:33:31] <elukey>	 vgutierrez: I tried your command but I see the same error that I find in the vk logs
[09:33:34] <elukey>	 verify return:1
[09:33:37] <elukey>	 140590443332928:error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected message:../ssl/statem/statem_clnt.c:393:
[09:33:43] <vgutierrez>	 elukey: errr
[09:33:53] <vgutierrez>	 I'm running that from cp4037 right now
[09:33:58] <elukey>	 same
[09:34:19] <jbond>	 works here
[09:34:30] <vgutierrez>	 https://www.irccloud.com/pastebin/wi5whc9i/
[09:35:06] <elukey>	 vgutierrez: line 9
[09:35:56] <vgutierrez>	 elukey: hmmm
[09:36:09] <vgutierrez>	 -state says that the server side is sending an unexpected message
[09:36:23] <vgutierrez>	 after the handshake is almost done
[09:36:28] <vgutierrez>	 https://www.irccloud.com/pastebin/o6cwlx9K/
[09:37:00] <elukey>	 I see yes.. maybe it is on the broker side the issue
[09:38:09] <jbond>	 vgutierrez: im also a bit confused that the Shared Requested Signature Algorithms also includes ECDSA+SHA512
[09:38:18] <vgutierrez>	 elukey: client config seems to be OK as long as you use ECDSA+SHA256
[09:41:01] <vgutierrez>	 jbond: me too TBH
[09:41:09] <elukey>	 the cergen's cert has Signature Algorithm: sha256WithRSAEncryption, and it works fine..
[09:41:24] <vgutierrez>	 jbond: I think those are the ones offered by the server
[09:41:35] <elukey>	 and the new one Signature Algorithm: ecdsa-with-SHA512
[09:42:30] <elukey>	 at this point it maybe be good to revert and repool cp4037, and possibly add a Varnishkafka canary that pushes to kafka-test somewhere
[09:42:46] <elukey>	 so if I break it no big deal, and in the future we'll be able to test changes inthere
[09:43:24] <jbond>	 SGTM
[09:43:47] <vgutierrez>	 jbond: so the requested ones seems the one being sent as part of the ServerHello MSG
[09:45:05] <elukey>	 all right reverting
[09:45:47] <vgutierrez>	 ack
[09:46:01] <elukey>	 thanks a lot to both for the help!
[09:47:33] <jbond>	 vgutierrez: thats what i though so i figured using ` -sigalgs ECDSA+SHA512` via the cli it would have worked.  but actully as i l;ook at it you get a much stranger error message in that case
[09:47:40] <jbond>	 https://phabricator.wikimedia.org/P49399 
[09:47:48] <jbond>	 no peer certificate available
[09:48:06] <vgutierrez>	 that's the handshake failing altogether
[09:48:07] * jbond a bit out of their TLS depth
[09:48:21] <vgutierrez>	 you get the same if you ask for tls 1.3
[09:48:35] <jbond>	 ahh could it be that ECDSA+SHA512 implies TLS1.3
[09:49:24] * jbond found this https://groups.google.com/a/chromium.org/g/net-dev/c/A-LcSmj5TBE
[09:50:37] <jbond>	 seems to be similar bugs for chromium and mozila
[09:51:15] <vgutierrez>	 so... the generated certs don't work with TLSv1.2?
[09:52:00] <jbond>	 id be suprised if thats the case and we didn;t discover untill now ...
[09:52:16] <vgutierrez>	 it's weird that OpenSSL returns a verification OK and the error at the same time
[09:52:21] <vgutierrez>	 SSL handshake has read 4428 bytes and written 1174 bytes
[09:52:21] <vgutierrez>	 Verification: OK
[09:52:27] <jbond>	 yes
[09:53:17] <vgutierrez>	 jbond: do you have in mind another endpoint using this kind of cert but a different SSL/TLS implementation?
[09:53:21] <vgutierrez>	 aka not JVM
[09:53:26] <vgutierrez>	 maybe the puppetmasters?
[09:53:29] <jbond>	 yes just checking debmonitor
[09:55:04] <jbond>	 vgutierrez: https://debmonitor.discovery.wmnet
[09:55:12] <jbond>	 with certs in /etc/debmonitor/ssl
[09:58:14] <jbond>	 vgutierrez: the following works 
[09:58:14] <jbond>	 sudo openssl s_client -connect debmonitor.discovery.wmnet:443 -tls1_2 -cipher ECDHE-ECDSA-AES256-GCM-SHA384 -cert /etc/debmonitor/ssl/debmonitor__sretest1001_eqiad_wmnet.pem -key /etc/debmonitor/ssl/debmonitor__sretest1001_eqiad_wmnet-key.pem -chainCAfile /etc/ssl/certs/wmf-ca-certificates.crt
[09:58:33] <jbond>	 anything < tls1.2 is disabled server side 
[09:59:07] <jbond>	 adding -sigalgs ECDSA+SHA512
[09:59:09] <jbond>	 also works
[09:59:27] * jbond tempted to blame java
[10:01:03] <elukey>	 librdkafka could also play a role, not sure
[10:01:36] <vgutierrez>	 elukey: librdkafka uses OpenSSL
[10:01:45] <vgutierrez>	 not at fault here IMHO
[10:01:58] <elukey>	 sure sure
[10:02:31] <elukey>	 we could see if the issue repro also on kafka-test1006.eqiad.wmnet:9093, restart the kafka broker with max tls/ssl logs for the jvm and see why it complains
[10:02:40] <elukey>	 this is the only other thing that I can think of
[10:02:59] <vgutierrez>	 sounds good
[10:07:24] <jbond>	 +1
[10:25:23] <vgutierrez>	 >>> ??? [length 0005]
[10:25:24] <vgutierrez>	     15 03 03 00 1a
[10:25:24] <vgutierrez>	 >>> TLS 1.2, Alert [length 0002], fatal unexpected_message
[10:25:24] <vgutierrez>	     02 0a
[10:25:30] <vgutierrez>	 that 15 03 03 00 1a is the unexpected message
[10:25:59] <vgutierrez>	 per RFC 5246 that's a certificate_verify (0x15) message
[10:26:45] <vgutierrez>	 same RFC..
[10:26:49] <vgutierrez>	 https://www.irccloud.com/pastebin/HY3EbPUd/
[10:28:39] <vgutierrez>	 hmm forget it
[10:28:42] <vgutierrez>	 15 != 0x15 :)
[10:33:40] <vgutierrez>	 elukey: so it's definitely related to mTLS
[10:35:06] <vgutierrez>	 elukey: are we sure that JVM 8 supports ecdsa-with-SHA512?
[10:36:32] <vgutierrez>	 elukey: https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#Signature
[10:36:35] <vgutierrez>	 seems to be listed there
[10:37:50] <vgutierrez>	 elukey: maybe our security config file disables it?
[10:37:57] <vgutierrez>	 as moritzm mentioned before
[11:09:27] <wikibugs>	 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10User-aborrero: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero)
[11:12:08] <wikibugs>	 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10User-aborrero: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero)
[11:13:12] <wikibugs>	 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10User-aborrero: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero) p:05Triage→03Medium
[12:27:21] <wikibugs>	 10Traffic, 10Infrastructure-Foundations, 10SRE-tools: Write a cookbook to roll reboot cache hosts - https://phabricator.wikimedia.org/T338783 (10Volans) p:05Triage→03Medium
[12:28:06] <elukey>	 vgutierrez: possibly yes, I tried to check in the security config file for java, but didn't find anything that may lead to ecdsa-sha512 not usable
[12:29:50] <elukey>	 vgutierrez: https://phabricator.wikimedia.org/P49400
[12:46:17] <vgutierrez>	 elukey: could we run something like https://stackoverflow.com/a/9334357 :?
[13:03:18] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:rack/setup/install dns100[456] - https://phabricator.wikimedia.org/T326685 (10Papaul) @ssingh any update on this?
[13:04:36] <elukey>	 vgutierrez: I can try but my java is a little rusty :D
[13:05:01] <elukey>	 I am going to do some tests with debug logging in kafka test, hopefully we'll get more info
[13:07:24] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:rack/setup/install dns100[456] - https://phabricator.wikimedia.org/T326685 (10ssingh) Hi @papaul: We are all good from dc-ops side, this is on Traffic now. We wanted to get a few NTP changes out of the way before reimaging the next batch and therefore it's blocke...
[13:17:56] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:rack/setup/install dns100[456] - https://phabricator.wikimedia.org/T326685 (10Papaul) @ssingh thanks
[13:29:22] <wikibugs>	 10Traffic, 10SRE: Q4:rack/setup/install dns100[456] - https://phabricator.wikimedia.org/T326685 (10ssingh)
[13:32:12] <elukey>	 so in the debug logs I see
[13:32:13] <elukey>	 Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[13:37:57] <elukey>	 and I also found this 
[13:37:57] <elukey>	         # https://phabricator.wikimedia.org/T182993#4208208
[13:37:58] <elukey>	         $ssl_java_opts = '-Djdk.tls.namedGroups=secp256r1 -XX:+UseAES -XX:+UseAESIntrinsics'
[13:41:07] <elukey>	 the cert should be good though
[13:41:07] <elukey>	                 ASN1 OID: prime256v1
[13:41:07] <elukey>	                 NIST CURVE: P-256
[13:57:09] <vgutierrez>	 elukey: err so it looks like the server side needs to get updated the CA file for the client verification
[13:57:41] * vgutierrez wondering...
[13:59:11] <vgutierrez>	 yeah, not even chained works
[13:59:21] <elukey>	 vgutierrez: I was reading that the cryptic error msg could indicate a generic tls handshake error, in theory the broker has the root PKI ca in its bundle and we send the chained cert
[14:04:25] <vgutierrez>	 -Djdk.tls.namedGroups=secp256r1
[14:04:37] <vgutierrez>	 so.. if that disables all the other eliptic curves
[14:04:48] <vgutierrez>	 we got a problem
[14:04:59] <vgutierrez>	 cause the intermediate CA pub key uses secp521r1
[14:05:17] <vgutierrez>	 https://www.irccloud.com/pastebin/J0MK4Ate/
[14:05:40] <vgutierrez>	 funny choice BTW :)
[14:07:22] <vgutierrez>	 according to the documentation that's the case
[14:07:46] <vgutierrez>	 from https://www.java.com/en/configure_crypto.html, it says that jdk.tls.namedGroups can be used to "Disable weak named curves by default in TLS, CertPath, and Signed JARs"
[14:08:26] <vgutierrez>	 elukey: it looks like secp521r1 needs to be added there as well
[14:08:35] <elukey>	 I tried to remove the setting but didn't solve the issue, do you think that we should add secp521r1?
[14:10:08] <vgutierrez>	 so secp521r1 needs to be theer
[14:10:10] <vgutierrez>	 *there
[14:10:22] <vgutierrez>	 it's used both on the intermediate CA and the root CA
[14:10:58] <vgutierrez>	 elukey: what do we have on jdk.disabled.namedCurves :?
[14:12:15] <elukey>	 vgutierrez: nothing afaics, it is set only for jdk11
[14:12:58] <elukey>	 ah no wait
[14:13:35] <vgutierrez>	 https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html lists secp521r1 as one of the recommended curves
[14:13:47] <elukey>	 yeah I see it only for java 11
[14:13:53] <vgutierrez>	 along with secp256r1 and secp384r1
[14:13:57] <elukey>	 the disabled etc..
[14:14:13] <vgutierrez>	 so as long as we aren't actually disabling it, we should be ok in that front
[14:14:52] <vgutierrez>	 same with ecdsa-with-SHA512 to be able to validate the signature
[14:17:53] <vgutierrez>	 BTW SHA512withECDSA support is provided by  $JAVA_HOME/lib/libsunec.so according to https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html
[14:18:28] <vgutierrez>	 we definitely need to be able to list the supported algorithms in runtime
[14:18:35] <vgutierrez>	 it's gonna be way easier :)
[14:30:53] <vgutierrez>	 https://www.irccloud.com/pastebin/EI8WD0qw/
[14:31:02] <vgutierrez>	 elukey: ^^ that's a naive test
[14:31:15] <vgutierrez>	 elukey: do you have the list of all the runtime JVM options that we are using?
[14:32:11] <elukey>	 vgutierrez: ack so IIUC SHA512withECDSA is supported right?
[14:32:24] <elukey>	 vgutierrez: the best it to grep for kafka on the node
[14:32:32] <elukey>	 err on ps -aux on the node
[14:32:42] <elukey>	 because the options are added in multiple places IIRC
[14:32:50] <elukey>	 most of them are in /etc/default/kafka though
[14:32:50] <vgutierrez>	 got it
[14:35:47] <vgutierrez>	 nothging related to tls or java.security besides the one already appended
[14:36:19] <vgutierrez>	 also a quick check with keytool -v -list -keystore /etc/ssl/localcerts/wmf-java-cacerts shows that the root CA is in there since March 21st and systemctl shows that kafka was started on March 29th
[14:36:21] * vgutierrez running out of ideas
[14:37:45] <elukey>	 yeah me too, really weird
[14:38:22] <vgutierrez>	 elukey: but wait.. mTLS CA store could be different to ssl.truststore.location
[14:41:20] <vgutierrez>	 nah.. it's the only one configured in there
[14:41:25] <vgutierrez>	 so it must be it
[14:41:39] <wikibugs>	 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero)
[14:52:40] <vgutierrez>	 https://www.irccloud.com/pastebin/Nyw0vX6X/
[14:53:31] <vgutierrez>	 let's see if I can perform that using java
[15:07:30] <vgutierrez>	 hmm interesting
[15:07:40] <vgutierrez>	 it's failing to validate it with Java
[15:15:21] <vgutierrez>	 fixed my code.. at least the chain gets validated as expected..
[15:15:24] <vgutierrez>	 sigh
[15:18:24] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Epic: [tracking] Don't keep on the public vlans hosts that don't require it - https://phabricator.wikimedia.org/T317177 (10aborrero)
[15:18:34] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10Papaul)
[15:18:40] <elukey>	 just to have some common ground - varnishkafka works if we don't set the sigalgs setting, so I am wondering what settings it uses 
[15:18:42] <wikibugs>	 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) 05Open→03Stalled This is done for codfw1dev DNS servers.  I'll mark this task as stalled until we work on...
[15:18:55] <elukey>	 because so far we didn't manage to repro a good handshake with the new cert/key right?
[15:20:13] <vgutierrez>	 elukey: with openssl? nope
[15:20:42] <vgutierrez>	 elukey: wait? what? without sigalgs work?
[15:21:18] <elukey>	 vgutierrez: it didn't emit any error log, I haven't tested sending a request, but yes it worked fine
[15:21:46] <elukey>	 this was the original point that I made, but we investigate so many things 
[15:23:09] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10Papaul)
[15:28:50] <vgutierrez>	 elukey: got it working now with OpenSSL
[15:29:17] <elukey>	 \o/
[15:29:21] <elukey>	 what magic did you use?
[15:29:28] <vgutierrez>	 elukey: -cert_chain :)
[15:29:33] <vgutierrez>	 https://www.irccloud.com/pastebin/0a2OM0CN/
[15:30:05] <vgutierrez>	 you can see that ECDSA+SHA256 is in there
[15:30:18] <vgutierrez>	 so.. for some reason varnishkakfa isn't feeding the intermediate CA as expected
[15:30:23] <vgutierrez>	 *varnishkafka even
[15:32:56] <vgutierrez>	 If I don't force any sigalg it still uses ECDSA+SHA256
[15:32:58] <wikibugs>	 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero)
[15:33:28] <vgutierrez>	 so.. know we know that kafka-jumbo is able to validate the cert as expected using another SSL client
[15:33:36] <vgutierrez>	 *now we know (damn brain)
[15:33:42] <elukey>	 I am checking the ssl. options in https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html but I don't find an equivalent for -cert_chain
[15:34:04] <vgutierrez>	 elukey: do we have an easy way of capturing the TLS handshake?
[15:35:08] <vgutierrez>	 elukey: that's because https://github.com/confluentinc/librdkafka/blob/master/src/rdkafka_ssl.c#L1177-L1178
[15:35:11] <elukey>	 vgutierrez: if you check on kafka-test1006 with `sudo journalctl -u kafka -f` you should see the ssl handshake debug logging
[15:35:22] * vgutierrez checking
[15:36:01] <vgutierrez>	 that's some verbosity
[15:36:09] <elukey>	 you asked for it :D
[15:37:00] <vgutierrez>	 not sure the TLS data is there anymore
[15:37:56] <elukey>	 ah yes lemme fix it
[15:38:07] <elukey>	 vgutierrez: now you have it
[15:38:58] <vgutierrez>	 that's from a varnishkafka producer,.right?
[15:39:05] <vgutierrez>	 the one starting at 15:37:57
[15:39:09] <elukey>	 openssl
[15:39:16] <vgutierrez>	 but that's working
[15:39:16] <elukey>	 did it manually
[15:39:18] * vgutierrez cries
[15:39:27] <elukey>	  sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[15:39:31] <elukey>	 nope it doesn't
[15:39:42] <vgutierrez>	 elukey: did you use the proper CLI?
[15:39:51] <elukey>	 ah ok no if you see other ones those are the other brokers probably
[15:40:03] <elukey>	 they do communicate via TLS as well
[15:40:33] <elukey>	 try to send a request to it via openssl and you'll see the data
[15:41:12] <vgutierrez>	 kafka-test1006:9093 right?
[15:41:15] <elukey>	 yeah
[15:42:29] <elukey>	 I also just noticed the .chain vs .chained in /etc/varnishkafka/ssl
[15:42:43] <vgutierrez>	 elukey: yup.. works like a charm with openssl
[15:42:45] <vgutierrez>	 no errors at all
[15:42:57] <elukey>	 vgutierrez: sure if you use the cert_chain right?
[15:43:04] <vgutierrez>	 that's how it should be :)
[15:43:48] <vgutierrez>	 elukey: I'm guessing that we the "old PKI" we didn't have an intermediate CA?
[15:44:11] <elukey>	 old pki namely puppet ca? yes :D
[15:44:33] <vgutierrez>	 yes
[15:44:41] * vgutierrez checking another thing..
[15:45:20] <elukey>	 vgutierrez: could it be that maybe I used the .chained pem file (containing separated charts) vs the .chain one (containing only one)?
[15:45:23] <vgutierrez>	 elukey: so.. it looks like .chained is wrong
[15:45:29] <elukey>	 exactly :D
[15:45:43] <vgutierrez>	 per SSL_CTX_use_certificate_chain_file() documentation
[15:45:54] <vgutierrez>	 The certificates must be in PEM format and must be sorted starting with the subject's certificate (actual client or server certificate), followed by intermediate CA certificates if applicable
[15:46:57] <vgutierrez>	 but on the .chained.pem file the first certificate is the intermediate CA one
[15:47:04] <elukey>	 oooof
[15:47:13] <vgutierrez>	 we should fix that one and see what happens
[15:47:19] * vgutierrez wants a beer if that solves the issue
[15:47:27] <elukey>	 ok so the "chained" one is the right one, but not in the right format
[15:47:39] <vgutierrez>	 at least not according to OpenSSL documentation
[15:47:48] <vgutierrez>	 maybe behavior != documentation
[15:47:59] <vgutierrez>	 but at the very least the file doesn't match docs behavior
[15:51:10] <elukey>	 vgutierrez: to avoid loosing all the context, do you mind to add a quick summary to https://phabricator.wikimedia.org/T337825 ?
[15:54:06] <wikibugs>	 10Traffic: Create a cookbook to reboot CDN hosts - https://phabricator.wikimedia.org/T338813 (10BCornwall)
[15:55:40] <wikibugs>	 10Traffic: Create a cookbook to reboot CDN hosts - https://phabricator.wikimedia.org/T338813 (10BCornwall) 05Open→03In progress p:05Triage→03Medium
[15:56:57] <vgutierrez>	 elukey: https://phabricator.wikimedia.org/T337825#8923598
[15:57:04] <wikibugs>	 10Traffic, 10Data-Engineering, 10Data-Platform-SRE, 10SRE: Move varnishkafka to PKI - https://phabricator.wikimedia.org/T337825 (10Vgutierrez) I've replicated a successful mTLS handshake with openssl s_client using the following CMD: ` vgutierrez@cp4037:~$ sudo openssl s_client -connect kafka-jumbo1001.eqi...
[16:03:02] <elukey>	 vgutierrez: <3
[16:03:19] <vgutierrez>	 6pm already??
[16:03:24] <vgutierrez>	 OMG, I got nerd snipped big time
[16:03:31] <elukey>	 I know I nerd sniped you badly today
[16:05:06] <vgutierrez>	 elukey: let me know if fixing the .chained order fixes the issue
[16:05:42] <vgutierrez>	 it should be "as easy" as disabling puppet on cp4037 and "re-ordering" the file
[16:05:55] <elukey>	 I can try to run a vk test instance on cp4037 that only emits to stdout
[16:05:58] <elukey>	 will try to do it
[16:06:51] <elukey>	 ah no of course I need to set kafka stuff sigh
[16:12:35] <wikibugs>	 10Traffic, 10SRE, 10Puppet, 10User-jbond: In valid byte sequence: File[/etc/update-ocsp.d/hooks/trafficserver-tls-ocsp] - https://phabricator.wikimedia.org/T238198 (10jbond)
[16:12:39] <vgutierrez>	 elukey: let's try tomorrow morning then :)
[16:14:06] <wikibugs>	 10Traffic, 10SRE, 10Puppet, 10User-jbond: In valid byte sequence: File[/etc/update-ocsp.d/hooks/trafficserver-tls-ocsp] - https://phabricator.wikimedia.org/T238198 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez @jbond I think we can close this one
[16:36:55] <elukey>	 vgutierrez: something interesting - it works when I use kafka test as target :D
[16:37:03] <elukey>	 the regular chained file I mean
[16:37:39] <elukey>	 but the broker runs with
[16:37:40] <elukey>	 -Djdk.tls.namedGroups="secp256r1,secp521r1"
[16:40:49] <vgutierrez>	 elukey: but I'm able to get a succesful tls handshake against jumbo1001 with openssl :|
[16:40:57] <vgutierrez>	 with just secp256r1
[16:41:29] <elukey>	 yes sorry same thing for test, I just fixed it
[16:41:31] <elukey>	 what the hell
[16:41:41] <elukey>	 now I am very confused
[16:42:21] <bblack>	 yeah nobody really needs or wants 521
[16:42:32] <vgutierrez>	 bblack: we do apparently :)
[16:42:36] <bblack>	 :P
[16:43:07] <vgutierrez>	 elukey: are we sure that kakfa-jumbo is using the current contents of the truststore and not a deprecated version?
[16:43:25] <vgutierrez>	 sure like 100% bet the beer comsumption of a whole year sure?
[16:44:26] <elukey>	 vgutierrez: I'd say so, otherwise I wouldn't have moved all brokers to PKI
[16:44:43] <jbond>	 vgutierrez: can you raise a ticket about not using secp521r1 (it will be hard to change buyt we can try)
[16:45:14] <jbond>	 also one on the ordering.  AFAIk the order hasd not caused issue with openssl before.  however i have noticed that golang is strict about this but hadn;t realised it was a spec thing
[16:45:17] <elukey>	 in the meantime I filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/929384/ to fix the path
[16:45:39] <elukey>	 jbond: I tried with the different order but even openssl seems to complain
[16:45:44] <elukey>	 key mismatch it says
[16:46:09] <vgutierrez>	 jbond: stuff like https://bugs.chromium.org/p/chromium/issues/detail?id=478225 :?
[16:47:27] <vgutierrez>	 elukey: openssl s_client -cert doesn't expect the chain to be embedded there at all AFAIK
[16:47:27] <elukey>	 anyway, we can call it a day, I'll restart tomorrow
[16:47:35] <jbond>	 elukey: yes as said im not sure its the solution to this issue but it should still be fixed as we have seen an issue with go
[16:47:37] <elukey>	 have a good evening folks!
[16:47:48] * vgutierrez follows elukey lead
[16:59:11] <wikibugs>	 10Traffic, 10serviceops, 10Datacenter-Switchover, 10Patch-For-Review: Figure out what changes are needed in the traffic layer for having codfw be the r/w DC for half a year - https://phabricator.wikimedia.org/T337535 (10BBlack) The more I've thought about this issue, I think we should probably stick with t...
[17:07:50] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10Papaul)
[17:08:25] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10Papaul)
[17:25:16] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10Papaul)
[17:28:18] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10Papaul)
[17:28:24] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, 10User-jbond: Investigate improvements to how puppet manages network interfaces - https://phabricator.wikimedia.org/T234207 (10jbond)
[20:04:54] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Codfw:row A/B: rack/cable new switches - https://phabricator.wikimedia.org/T332180 (10Papaul)