[04:46:35] <ryankemper>	 dcausse: I haven't deployed WDQS `0.3.104` because I wasn't sure if doing so might screw things up given the rollback to a previous checkpoint. I can deploy it tomorrow around the time of the weds meeting if we need to deploy wdqs
[05:13:50] <ryankemper>	 Incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/2022-02-22_wdqs_updater_codfw
[07:42:22] <gehel>	 ryankemper: thanks for the report!
[07:42:55] <gehel>	 dcausse: codfw is still depooled, I'm waiting for your confirmation before repooling
[08:03:19] <gehel>	 dcausse / ebernhardson: is there anything left to do on T293462 ? The last patches seem to be about cleanup (which is great!), but I'm not clear if there is more work to be done.
[08:03:19] <stashbot>	 T293462: Add user blocking in WCQS - https://phabricator.wikimedia.org/T293462
[09:01:55] <gehel>	 Physiotherapy with Oscar, back in 1h
[09:07:42] <dcausse>	 will try to reproduce the produce in yarn so that we can repool codfw soon
[09:27:59] <itamarWMDE>	 Hello search team,  I want to follow up on the codfw incident from yesterdy, as there is a patch by addshore reverting changes done to mitigate the temporary loss of codfw. IS codfw back up and running?
[09:29:26] <dcausse>	 itamarWMDE: yes it's back up but we'd like to keep down a little to debug something
[09:40:59] <dcausse>	 itamarWMDE: I'll comment on https://gerrit.wikimedia.org/r/c/operations/puppet/+/764830 once we're OK if you're fine with this
[10:19:46] <itamarWMDE>	 dcausse: ofc, thank you for the quick response!
[10:53:15] <gehel>	 lunch
[11:24:11] <dcausse>	 lunch
[12:50:13] <gehel>	 ejoseph: I might be a few minutes late for our meeting, the plumber is still trying to fix the toilets
[12:50:41] <ejoseph>	 alright
[12:51:52] <gehel>	 ejoseph: and he is done ! I'll be there on time !
[13:38:49] <dcausse>	 gehel: about T293462 I think remaining thing was to double check that the user_text is propagated to the query logs and then a cleanup indeed
[13:38:50] <stashbot>	 T293462: Add user blocking in WCQS - https://phabricator.wikimedia.org/T293462
[13:42:33] <dcausse>	 ryankemper: it's perfectly fine to deploy 0.3.104 to wdqs too
[14:03:14] <inflatador>	 Greetings
[14:04:11] <dcausse>	 o/
[15:00:00] <zpapierski>	 o/
[15:07:35] <dcausse>	 o/
[15:16:09] <ejoseph>	 dcausse: can we continue
[15:16:27] <dcausse>	 ejoseph: troubleshooting something at the moment
[15:16:43] <dcausse>	 ejoseph: did you find the log4j.properties file?
[15:16:56] <ejoseph>	 Yh
[15:17:11] <ejoseph>	 It seems correct
[15:17:36] <dcausse>	 it should say in which file the deprecation warnings are being logged
[15:17:53] <ejoseph>	 And also tried different queries but couldn’t trigger the deprecation error
[15:19:54] <dcausse>	 ejoseph: deprecation warnings, I think, are in another log file (not elasctisearch.log)
[15:45:40] <ejoseph>	 Oh
[15:52:31] <dcausse>	 can start the flink app in yarn but not in k8s...
[17:23:45] <inflatador>	 workout, back in ~30
[18:03:13] <inflatador>	 and back
[19:10:37] <inflatador>	 lunch, back in ~45
[19:41:38] <ebernhardson>	 fwiw, poking at some yarn worker nodes they hava a /usr/lib/jvm/java-11-openjdk-amd64 directory
[19:42:10] <ebernhardson>	 i dunno how to convince flink in yarn to use it to test if that triggers the load error
[19:50:36] <inflatador>	 aaand back
[19:59:29] <dcausse>	 looks like there's -yD yarn.taskmanager.env.JAVA_HOME
[19:59:37] <dcausse>	 I could try that perhaps
[20:00:30] * ebernhardson uses the ugliest hacks to get that info..something like spark.range(1).map(lambda x: subprocess.check_output('ls -l ...')).collect()
[20:00:47] <dcausse>	 :)
[20:00:49] <ebernhardson>	 seems worth trying perhaps, but try tomorrow :)
[20:32:31] <gehel>	 ryankemper: https://meet.google.com/aib-canx-cgh
[21:41:45] <inflatador>	 does anyone know if the instances in the WMCS "search" project have proper DNS names (and if so, what they are)?
[21:42:07] <taavi>	 <instancename>.<project>.eqiad1.wikimedia.cloud applies to all projects
[21:43:18] <inflatador>	 excellent! Thanks taavi 
[21:51:10] <ebernhardson>	 names get longer and longer :P i'm sad on instances where foo.eqiad.wmflabs stops working 
[21:51:28] <ebernhardson>	 i'll have to find some way to tab complete instance names or something
[21:52:55] <inflatador>	 yeah, maybe we could scrape Designate or something? taavi may have a better idea
[21:54:44] <taavi>	 you could do something like https://wikitech.wikimedia.org/wiki/User:Razzi/ssh_single_letter_domain_shortcut for the projects you use the most
[21:55:08] <taavi>	 for tab completion I don't really have any ideas for most projects
[21:55:18] <ebernhardson>	 thats a nice trick, fixes the length problem
[21:56:14] <inflatador>	 yeah, this is actually quite helpful
[22:53:12] * ebernhardson notices that super_detect_noop does a weird thing, where it can noop some updates to the document but let others through, so you end up paying the full update cost but still only update some values
[22:53:51] <ebernhardson>	 essentially it noops per-field, instead of deciding if it will update the document then applying all updates