[11:24:47] <dcausse>	 lunch
[13:03:19] <inflatador>	 <o/
[13:19:20] <dcausse>	 o/
[13:31:56] <ebernhardson>	 \o
[13:33:03] <dcausse>	 o/
[13:52:24] <inflatador>	 .o/
[14:28:15] <ebernhardson>	 opensearch-eqiad is currently scaling up, will do codfw next
[14:35:51] <ebernhardson>	 inflatador: not sure if it's relatd to the problems from before, but a pod came up on dse-k8s-worker1013.eqiad.wmnet and is showing the same networking problems we saw before
[14:36:01] <ebernhardson>	 as in, it can't find the cluster
[14:48:14] <Trey314159>	 Does anyone feel a strong need to retro today?
[14:49:11] <ebernhardson>	 not really
[14:54:17] <dcausse>	 same
[14:54:45] <ebernhardson>	 dcausse: oh, i found the mjolnir problem yesterday...it's the same problem as glent. Should have checked more carefully for other occurances: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2075/diffs
[14:55:25] <ebernhardson>	 unix_timestamp returned null, it propagated it's way through and made session id's null
[14:56:25] <dcausse>	 :/
[14:56:27] <dcausse>	 looking
[14:57:05] * ebernhardson sometimes wishes sql-like things were less null-safe and blew up instead of passing nulls through everything
[14:57:29] <dcausse>	 it's when we moved to spark-sql instead of hql?
[14:57:46] <ebernhardson>	 dcausse: no, the events started emitting millisecond precision on timestamps, but that was declaring a specific timestamp format
[14:58:21] <dcausse>	 ah ok, yes strict date parsing like this could not work well :/
[14:58:24] <ebernhardson>	 cast'ing avoids that since it doesn't require a specific format, it accepts a variety of formats
[15:00:12] <ebernhardson>	 will have to check source data, hopefully we can re-run the past few months
[15:04:32] <dcausse>	 sure
[15:05:43] <ebernhardson>	 opensearch-semantic-search-eqiad is now scaled up, 16 pods, ~600GB of memory, ~2.8TB disk.  I'm still restarting masters to pick up the new settings, but it should be good for indexing
[15:22:08] <inflatador>	 {◕ ◡ ◕}
[15:27:07] <ebernhardson>	 all restarted, doing codfw now
[15:49:58] <ebernhardson>	 codfw done now too, both clusters should be ready for loading data
[15:56:19] <dcausse>	 thanks!
[15:56:57] <dcausse>	 I could start shipping pt on both
[16:08:27] <ebernhardson>	 hmm, i tried to reset query_clicks_hourly tasks via kubectl exec...and all the pods restarted :S
[16:08:31] <ebernhardson>	 maybe it was coincidence...
[16:11:04] <ebernhardson>	 well, worked the second time 
[18:46:29] <dcausse>	 dinner
[19:07:39] <ebernhardson>	 almost amazing, checkout out opensearch 2.19 branch and ran `./gradlew clean`...took 4m31s
[19:07:45] <ebernhardson>	 s/checkout/checked/
[19:08:06] <inflatador>	 I hate to ask, but is that good or bad?
[19:13:07] <ebernhardson>	 thats bad :P  Most things it takes a couple seconds, but opensearch (and elasticsearch) have deep integration with the build system
[19:15:39] <inflatador>	 (╯°□°）╯︵ ┻━┻