[07:14:14] <gmodena>	 o/
[08:11:36] <dcausse>	 o/
[10:38:27] <gmodena>	 dcausse when you have a moment: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1047
[10:38:49] <gmodena>	 it fixes a DAG import error on the search instance (my fault)
[10:39:14] <dcausse>	 gmodena: looking
[10:39:27] <gmodena>	 dcausse thx!
[10:43:32] <dcausse>	 gmodena: +1, thanks for this! saw this error last week-end but decided to ignore it, it was not blocking anything and I knew you were working on something related :)
[10:44:09] <gmodena>	 dcausse ah! I thought it was introduced by a MR i merged earlier today
[10:44:27] <gmodena>	 same code path anyway (hopefully :D)
[10:44:46] <dcausse>	 sure, no worries! :)
[10:45:12] <dcausse>	 errand+lunch
[12:19:34] <gmodena>	 dcausse oof. Another patch in this saga (for when you have a moment): https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1049
[12:55:42] <dcausse>	 gmodena: approved, thanks!
[13:01:39] <gmodena>	  I was also just looking into mjolnir's failure on 2025-01-24
[13:02:02] <gmodena>	 Exception in thread "main" java.io.FileNotFoundException: File hdfs:/wmf/cache/artifacts/airflow/search/refinery-hive-0.2.55-shaded.jar does not exist.
[13:02:08] <gmodena>	 welp
[13:03:16] <gmodena>	 we declared that dep in artifacts.yaml 
[13:03:48] <gmodena>	 shouldn't it have auto-magically synced to hdfs?
[13:09:10] <gmodena>	 the release does exist https://archiva.wikimedia.org/repository/releases/org/wikimedia/analytics/refinery/hive/refinery-hive/0.2.55/
[13:09:15] <gmodena>	 I'll check in slack
[13:19:35] <gmodena>	 dcausse the dag import error is resolved
[13:19:44] <gmodena>	 errand+lunch
[13:19:47] <dcausse>	 \o/
[13:20:56] <dcausse>	 for the artifact if it's new we still need to run a manual scap to publish it, the airflow CD pipeline does not take care of these yet
[13:54:00] <gmodena>	 dcausse ack
[13:55:15] <gmodena>	 sync refinery-hive. I'll re-start mjolnir
[13:55:27] <gmodena>	 and the drop data job since I'm at it
[14:11:38] <inflatador>	 <o/
[14:13:42] <inflatador>	 dcausse sorry for the late notice, brouberol and I are talking opensearch migration in ~45m. I sent you an invite just now as optional...only if you want to join though
[14:13:57] <dcausse>	 o/
[14:14:13] <dcausse>	 inflatador: sure, I'll try to attend
[14:16:53] <inflatador>	 dcausse cool, can you see this doc? I used search platform email for perms https://docs.google.com/document/d/1ZFSDRMyzk4G929kbpJ-7bJGNCt87rW2YZKZHyAPekJk/edit?tab=t.0#heading=h.u4gqhcfnrjev
[14:17:46] <dcausse>	 inflatador: no I don't, saw the link on the search-platform email but this did not work well
[14:19:00] <inflatador>	 dcausse ACK, just added you as editor along w/the rest of the team
[14:19:05] <dcausse>	 thx!
[14:19:23] <gehel>	 inflatador: could you move that document in the team's shared drive? https://drive.google.com/drive/u/1/folders/1BtzlSjkUKz0Rdl-4mipzCTHG_H3QyRLS
[14:19:41] <gehel>	 It gives access to all of DPE automagically, and makes it easier to find.
[14:21:35] <inflatador>	 gehel ACK, done
[14:26:19] <inflatador>	 dcausse we'll probably discuss https://gerrit.wikimedia.org/r/c/operations/puppet/+/1090529 to begin with, feel free to look it over if you have time
[15:52:40] <dcausse>	 popularity_score is failing with "org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire 262144 bytes of memory, got 0", going to give a bit more to the workers...
[15:52:53] <dcausse>	 quick errand
[16:04:22] <gmodena>	 dcausse ack
[16:09:34] <gmodena>	 The refinery data cleaning script ran on the pod (hurray), but errored because its checksums have changed. I was afraid this would happen. We'll need a manual clean up, but I don't feel comfortable purging tables I'm not familiar with.
[16:21:51] <dcausse>	 gmodena: \o/, no worries about the manual cleanups, I'll take a look
[17:07:16] <inflatador>	 dcausse any objections to me merging the above Puppet patch today? Reimaging takes ~45m so I'd like to have that done before my pairing with Ryan later today. If not, happy to wait until right before our pairing session on Weds
[17:07:56] <dcausse>	 inflatador: no objections, please go ahead
[17:08:18] <inflatador>	 dcausse ACK, thanks
[17:08:21] <inflatador>	 workout, back in ~40
[17:41:55] <inflatador>	 back
[17:48:55] <dcausse>	 inflatador: quickly checking, do we have T379312 ?
[17:48:55] <stashbot>	 T379312: Release packages for opensearch 1.3.19 - https://phabricator.wikimedia.org/T379312
[17:50:06] <dcausse>	 I see the plugins: https://apt.wikimedia.org/wikimedia/pool/component/opensearch13/w/wmf-opensearch-search-plugins/
[17:57:16] <inflatador>	 dcausse ACK, checking no
[17:57:17] <inflatador>	 w
[17:57:36] <inflatador>	 I have not yet merged BTW
[18:00:41] <inflatador>	 according to debmonitor, the only hosts using opensearch 1 are datahubsearch (owned by DPE SRE). They are on 1.2.4, and so's the repo: https://apt-browser.toolforge.org/bullseye-wikimedia/thirdparty/opensearch1/
[18:10:24] <dcausse>	 we can use the "opensearch13" component I think to push opensearch 1.3.19 
[18:11:02] <dcausse>	 so that we don't messup with observability's version
[18:12:00] <inflatador>	 observability is on opensearch 2, just confirmed via https://debmonitor.wikimedia.org/ (not sure if you have access)? You can put a pkg name in there and it will tell you where it's installed
[18:12:37] <inflatador>	 anyway, I'll update that ticket and start building the 1.3.19 packages to publish to that component...it looks unused ATM
[18:38:33] <inflatador>	 lunch, back in ~40
[19:15:02] <inflatador>	 back
[19:49:45] <dcausse>	 gmodena: thanks for the link on https://airflow-search.wikimedia.org/variable/list/, I knew I was missing something but was not sure what! :) will do some testing tuning the mem settings there
[20:01:07] <inflatador>	 dr appointment, back in ~90
[20:08:36] <gmodena>	 dcausse np! 
[21:22:28] <inflatador>	 back