[09:31:17] <gehel>	 dcausse: we're in https://meet.google.com/nac-nzao-gpm with Leszek. Gabriele: if you want to have a last discussion about Search, feel free to join
[09:31:27] <dcausse>	 oops
[09:31:39] <gehel>	 dcausse: you're not late yet! (cc: gmodena)
[10:44:45] <dcausse>	 lunch
[11:31:33] <pfischer>	 gehel: I tried to pull a report from phabricator (using conduit) of things we’ve been working over the last quarter but apparently that is not a trivial task. How do you usually gather such report?
[11:36:49] <gehel>	 pfischer: mostly from memory, with some help from https://wikitech.wikimedia.org/wiki/Search_Platform/Weekly_Updates
[12:10:58] <gehel>	 pfischer: want to pair on preparing those slides?
[12:30:20] <pfischer>	 Sure, I have to pick up Luise from daycare but I’ll be back by 3:15
[12:31:09] <gehel>	 ping me when back!
[13:13:51] <inflatador>	   
[13:13:54] <inflatador>	 <o/
[13:15:06] <dcausse>	 o/
[13:20:13] <inflatador>	 I added rack/row awareness to relforge puppet code, but I'm guessing it doesn't take effect until after a service restart? checking...
[13:26:42] <pfischer>	 dcausse: Is anyone working on SUP at the moment? I see unstable warnings
[13:26:52] <dcausse>	 pfischer: it's me
[13:45:33] <dcausse>	 I'm into a weird state.... producer & consumer-search failing with "Caused by: java.io.IOException: Target file s3://cirrus-streaming-updater.wikikube-eqiad/consumer-search/checkpoints/1c39201d2b4f315e6efd4f5b56c03af4/chk-219389/_metadata already exists."
[13:46:30] <dcausse>	 suspended both jobs to get a better view
[13:54:06] <pfischer>	 dcausse: Shall we pair? Could we drop the state?
[13:54:41] <dcausse>	 pfischer: sure https://meet.google.com/aqs-cxnf-tor
[13:55:38] <inflatador>	 ryankemper https://phabricator.wikimedia.org/T380934 is unblocked if you wanna work on it when you get in
[13:56:30] <inflatador>	 I also just popped T390565 for the relforge decommission, but that will take a bit more work
[13:56:31] <stashbot>	 T390565: decommission relforge100[34] - https://phabricator.wikimedia.org/T390565
[14:09:32] <inflatador>	 I'm roll-restarting cloudelastic now to apply the new plugins...relforge will take awhile since it still has 1G hosts
[14:09:59] <inflatador>	 I'm banning them to decommission at the same time
[14:12:51] <ebernhardson>	 \o
[14:13:34] <dcausse>	 o/
[14:29:35] <gehel>	 Trey314159: chrome is crashing. Hopefully we were done...
[14:30:09] <Trey314159>	 gehel: I think so. good luck resurrecting chrome
[14:53:07] <inflatador>	 .o/
[14:55:52] <inflatador>	 Cloudelastic restart is done, so the vector search plugin should be available. I'm about to do relforge as well
[14:56:28] <ebernhardson>	 nice!  i still haven't managed to convince CI to build the image :S But i suspect it's something to do with the build host, going back to older blubber versions didn't help
[14:57:31] <ebernhardson>	 it has such a useless error message, failing very early in the build (looks like while reading the dockerfile): error: failed to solve: exit code: 2`
[14:57:53] <inflatador>	 booo
[14:58:17] <inflatador>	 if you still need help this afternoon LMK. I doubt I'd be much use, but happy to take a look
[14:58:26] <pfischer>	 dcausse: switching to savepoints as upgrade mode is one of the recommendations after schema changes
[14:59:57] <dcausse>	 yes I think it might be safer to use that all the time
[15:00:38] <ebernhardson>	 my best guess i the build hosts are doing something wrong, maybe perms?  The next message in the log should have been about transfering the local context into the build (which historicallly was 2 bytes, so basically nothing)
[15:01:52] <dcausse>	 :/
[15:02:28] <ebernhardson>	 i should probably just file a ticket, all the info i could find about this basically says it's a builtctl problem, maybe invocation args or permissions
[15:02:51] <ebernhardson>	 but thats all done by kokkuri
[15:04:55] <inflatador>	 yeah, sounds like a ticket for relforge is probably the best
[15:05:43] <ebernhardson>	 oh not relforge, this is our docker image failing to build and publish in CI.  It builds fine in test, but then when trying to publish an image to the docker-registry it fails to even start the build
[15:06:10] <inflatador>	 sorry...I meant to say 'releng'
[15:43:36] <inflatador>	 relforge restart is finished
[15:59:29] <ebernhardson>	 curiously, reverting all the way back to blubber 0.16.0 let the image build. 1.3.20-5 is on docker-registry.wm
[16:01:39] <dcausse>	 weird...
[16:03:12] <ottomata>	 dcausse: pfischer ! from staff meeting.  "modernized FLink job with wikimedia-event-utilities". Is WDQS using event-utilities with the EventRowTypeInfo stuff? 
[16:03:22] <dcausse>	 ottomata: yes
[16:03:24] <ottomata>	 no more case classes?!  
[16:03:30] <dcausse>	 still some :)
[16:03:43] <ottomata>	 wow I did not know yall were doing that!  very cool!
[16:03:58] * ebernhardson is waiting for our LLM overlords to migrate the scala to java
[16:03:59] <ottomata>	 would love to peruse a patch or ticket, got a link?
[16:04:09] <dcausse>	 sure
[16:05:39] <dcausse>	 ottomata: mainly https://gerrit.wikimedia.org/r/q/topic:%22output-schema-v2%22
[16:21:46] <dcausse>	 ryankemper: for when you have time could you deploy https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/1112251?
[16:27:57] <dcausse>	 Trey314159: for some reasons I thought you had written some notes on the broken lucene boolean logic but can't find anything in your notes, do you have something about that somewhere?
[16:28:42] <ebernhardson>	 oddly, i also looked for that last week and didn't find it
[16:38:47] <dcausse>	 dinner
[16:47:50] <Trey314159>	 dcausse & ebernhardson: the broken boolean stuff is not in my notes, it's here: https://www.mediawiki.org/wiki/Help:CirrusSearch/Logical_operators
[16:54:44] <ebernhardson>	 ahh, i suppose that makes sense.  I was searching in the user namespaces with intitle filters
[17:05:58] * ebernhardson will someday remember there is a 'run puppet compiler' button before sending a manual `check experimental` code review
[17:32:12] <ebernhardson>	 curious graph, a low level of consistent cirrus failed requests since 12:50 yesterday.  Not sure its meaningful, but curious: https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1&from=now-2d&to=now&viewPanel=9
[17:36:53] <Trey314159>	 what the bot?
[17:38:58] <ebernhardson>	 perhaps, but bad queries should be classified as errors, failures should be internal things. 
[17:39:29] <ebernhardson>	 or i guess we call them "rejected" in the graph, not errors
[17:40:26] <ebernhardson>	 hmm, logstash has some not happy things: Elasticsearch response does not have any data. upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
[17:40:47] <ebernhardson>	 much higher than 2/s though :P
[17:41:30] <ebernhardson>	 oh, well actually maybe.  2286 messages in the last 15 minutes  which gives 2.5/s
[17:49:58] <ebernhardson>	 it all forwards through envoy, in a quick test of 10000 requests to the banner on each port (localhost:610{2,3,4}),  6102 failed 5 times, 6103 failed 4 times, 6104 failed 0 times.
[17:52:34] <ebernhardson>	 i suppose plausibly (hopefully!) envoy is reusing connections, so the ratios aren't quite right. The envoy dashboard gives a 10-20% connection fail rate
[17:52:49] <ebernhardson>	 looks like some host has a bad TLS cert, will see if can figure out which one
[17:53:10] <ebernhardson>	 envoy might know, but not sure how to find out :P poking
[17:55:42] <ebernhardson>	 i guess envoy doesn't know, it talks to LVS
[18:00:59] <ebernhardson>	 might have been 1068, i've just depooled it. Will see if the errors stop
[18:09:01] <ebernhardson>	 hmm, well 1068 is certainly failing tls, but depooling it didn't change the error rate :(
[18:10:56] <ebernhardson>	 annoyingly curl will print subject/start/expire date on a succesfull connection. But on a failed connection it just say "bad, no connection for u
[18:11:05] <ebernhardson>	 s/failed connection/expired cert
[18:12:14] <ebernhardson>	 inflatador: not sure what to do with it, elastic1068 has an expired TLS cert: https://phabricator.wikimedia.org/P74506
[18:12:34] <ebernhardson>	 i ran `depool` on elastic1068, but i'm still able to connect to it when using https://search.svc.eqiad.wmnet:9243/ (just repeat enough times till it fails)
[18:13:05] <ebernhardson>	 test with: curl -vvkI https://search.svc.eqiad.wmnet:9443/ 2>&1 | grep -e 'subject:' -e 'start date:' -e 'expire date:'
[18:15:59] <ebernhardson>	 shrug, ran the same depool command again. Maybe this time :P
[18:23:48] * ebernhardson realizes its because the right command is `sudo depool`, `depool <reason>` fails silently with no error message
[18:24:57] <ebernhardson>	 that does appear to have silenced the messages...will make a ticket to get the TLS fixed and the server repooled
[18:25:56] <dcausse>	 Trey314159: thanks for the link!
[18:26:16] <Trey314159>	 dcausse: sure thing!
[18:26:32] <dcausse>	 ebernhardson: good catch!
[18:27:30] <dcausse>	 I wonder if we got alerted by some other means for this cert expiration
[18:28:51] <ebernhardson>	 it's partially odd because the certs are only good for ~30 days, so there must be an auto-renew thing.  Also most of the other certs expire Apr 16, only this host expired Mar 30
[18:29:18] <ebernhardson>	 dcausse: while you're here, which months is your sabatical?
[18:29:26] <ebernhardson>	 thinking about summer vacations :)
[18:29:38] <dcausse>	 ebernhardson: june and july
[18:29:42] <ebernhardson>	 kk, thanks
[18:41:49] <ebernhardson>	 there is still a (much much lower) rate of "Elasticsearch response does not have any data. upstream connect error or disconnect/reset before headers. reset reason: connection failure"...not sure what to do with that
[18:42:02] <ebernhardson>	 feels like when the two sides disagree on keeping the connection open
[18:42:33] <ebernhardson>	 but more like 15/minute
[19:09:32] * ebernhardson tries to understand the difference between envoy TCP proxy idle_timeout (1hr), and TCP protocol idle_timeout (10min)
[19:17:40] <ebernhardson>	 i also wonder why keepalive is either not specified, or set to 4s, 4.5s, or 10s.  I suppose i was expecting more like 600s
[20:37:51] <inflatador>	 back
[20:39:42] <inflatador>	 ebernhardson thanks for reporting this. I guess we need to look at our health checks, IMHO we should automatically depool hosts under these circumstances
[20:41:37] <ebernhardson>	 inflatador: i thought so too, it's in the ticket AC :)
[20:41:48] <ebernhardson>	 although i guess i said "Consider" instead of a requirement
[20:42:52] <inflatador>	 yeah, I'm still waking up, but I guess SSL certificate verification doesn't cause an HTTP error...so that probably wouldn't be the load balancer's job. Hmm
[20:49:57] <inflatador>	 The cert on elastic1068 is current: `Not Before: Wed Mar 19 12:19:00 UTC 2025 Not After: Wed Apr 16 12:19:00 UTC 2025`. reloading nginx fixed the problem, but I'm pretty sure Puppet should be doing that
[20:56:52] <ebernhardson>	 hmm, thats curious. So it suggests nginx wasn't properly restarted when the certificate was refreshed. Maybe the puppet run failed early, then the next run didn't know to restart nginx?
[20:57:09] <ebernhardson>	 checking if puppet logs go that far back
[20:57:22] <inflatador>	 Yeah, that's my guess as well
[20:57:48] <ebernhardson>	 hmm, no, the oldest puppet log is Mar 24, 
[20:58:21] <inflatador>	 kinda surprised that we don't get SSL certificate checks "for free", but that may be a consequence of us using nginx for TLS termination. I think every other role besides cirrus is on envoy
[21:04:00] <ebernhardson>	 hmm, puppet ran at 11:53, then 12:23, can see the renew certificate ran. Says it scheduled refresh of Exec[nginx-reload]
[21:04:28] <ebernhardson>	 then it failed with: (/Stage[main]/Profile::Tlsproxy::Instance/File[/etc/systemd/system/nginx.service.d/security.conf]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/profile/tlsproxy/nginx-security.conf: Request to
[21:04:29] <ebernhardson>	 https://puppetserver1003.eqiad.wmnet:8140/puppet/v3/file_metadata/modules/profile/tlsproxy/nginx-security.conf?links=manage&checksum_type=sha256&source_permissions=ignore&environment=production failed after 0.002 seconds: Failed to open TCP connection to puppetserver1003.eqiad.wmnet: 8140 (Connection refused - connect(2) for "puppetserver1003.eqiad.wmnet" port 8140)
[21:04:57] <ebernhardson>	 which caused the rest to give an error about skipping because of failed dependencies
[21:05:15] <ebernhardson>	 (from syslog.12.gz)
[21:05:33] <ebernhardson>	 not sure what to do with that :P
[21:05:36] <inflatador>	 ah, we do have SSL certificate alerts BTW
[21:06:23] <ebernhardson>	 seems like it shuld have retried that connection a few times, intsead of failing once in 2ms
[21:07:31] <inflatador>	 And we did get an alert on Mar 27th, so I guess I need to pay more attention to those ;(
[21:12:47] <ebernhardson>	 seems like SSL cert issues should be fairly loud, maybe creating a ticket?
[21:28:06] <inflatador>	 I'd be OK with that. The bigger issue is that we have a lot of noisy alerts in IRC that drown out the important ones