[07:36:53] <gmodena>	 o/
[08:32:37] <dcausse>	 o/
[09:35:22] <gmodena>	 dcausse no rush: do you maybe have some swift magic (and credentials :D) to fetch the xgoboost models uploaded by mjolnir ?
[09:48:37] <dcausse>	 gmodena: looking
[10:44:16] <dcausse>	 gmodena: I believe it's in 'sudo -u analytics-search kerberos-run-command analytics-search hdfs dfs -text /user/analytics/swift_auth_analytics_admin.env'
[10:46:54] <dcausse>	 errand+lunch
[10:48:40] <dcausse>	 ignore the kerberos-run-command I think you can read this file with your own kerberos session
[10:49:44] <gmodena>	 dcausse ack. checking
[10:51:43] <gmodena>	 dcausse yep. works without sudo. fwiw I don't think I'm part of `search-analytics` anyway (modulo inherited membership)
[11:04:21] <gmodena>	 Ah! we also store the models in HDFS
[14:06:53] <dcausse>	 yes I think the swift part is only helping when publishing to elastic
[14:10:14] <cdanis>	 the swift copy is for clients that can't access HDFS directly yeah
[14:15:47] <inflatador>	 <o/
[14:21:37] <gmodena>	 o/
[14:22:39] <gmodena>	 dcausse cdanis ack
[14:23:04] <dcausse>	 o/
[14:30:10] <inflatador>	 dcausse we are looking at migrating Airflow-search instance to Kubernetes. Is now a good time, or do we need to schedule a maintenance window? Estimated downtime is 1-2 hours
[14:30:31] <dcausse>	 inflatador: looking
[14:30:38] <inflatador>	 https://airflow-search.wikimedia.org/dagrun/list/+
[14:30:39] <inflatador>	 https://airflow-search.wikimedia.org/dagrun/list/
[14:31:49] <dcausse>	 only one task is currently running https://airflow-search.wikimedia.org/taskinstance/list/?_flt_0_state=running#
[14:32:05] <dcausse>	 inflatador: I'd say yes
[14:33:14] <inflatador>	 dcausse yes, OK to go ahead or yes we should schedule a window?
[14:33:27] <dcausse>	 inflatador: ok to go ahead :)
[14:33:36] <inflatador>	 dcausse ACK, will give you a heads-up when we're starting
[14:33:42] <dcausse>	 ack
[14:58:58] <gmodena>	 dcausse thanks for T383333!
[14:58:59] <stashbot>	 T383333: Add gmodena to analytics-search-users - https://phabricator.wikimedia.org/T383333
[15:07:23] <dcausse>	 yw!
[16:48:57] <inflatador>	 For those not in the Slack thread, the airflow-search instance is now fully migrated to k8s
[16:51:31] <inflatador>	 dcausse for the categories stuff, do we need to apply all daily dumps when reloading, or just the weekly dump + one daily dump? Based on https://gerrit.wikimedia.org/r/plugins/gitiles/wikidata/query/rdf/+/refs/heads/master/dist/src/script/loadCategoryDaily.sh#20 I'm assuming only one daily dump?
[16:56:56] <dcausse>	 inflatador: I think loadCategoryDaily.sh just loads one day, so say the last weekly dump available is from monday and we run on friday it has to load the weekly from monday + the dailies for tue, wed, thur and fry
[16:57:34] <dcausse>	 s/fry/fri
[17:02:42] <inflatador>	 dcausse ACK. But based on my reading of the current script, it only seems to load the latest daily? Or am I missing something
[17:16:04] <inflatador>	 lunch, back in ~1h
[17:30:16] <dcausse>	 inflatador: yes loadCategoryDaily.sh loads only one day but since it's suppose to run every day it's ok
[17:30:57] <dcausse>	 I have no clue if there are mechanism to capture cases when a host is down for several days
[17:32:05] <dcausse>	 or even if the categories reload cookbook is taking care of populating days after importing the weekly dump
[17:33:09] <dcausse>	 but tbh I haven't looked at those scripts in a while and from my memory they're pretty much a big mess
[17:41:11] <dcausse>	 sigh... with superset you not only need perms on the dashboard to edit it but every single charts used by it...
[18:00:04] <btullis>	 There is a failing DAG on the airflow-search instance. Have you spotted it? https://airflow-search.wikimedia.org/dags/query_clicks_hourly/grid?dag_run_id=scheduled__2025-01-09T12%3A00%3A00%2B00%3A00&task_id=transform_hourly&base_date=2025-01-09T14%3A00%3A00%2B0000&tab=logs
[18:00:22] <dcausse>	 btullis: looking
[18:00:50] <btullis>	 It looks like it's trying to call the `hive` CLI directly, which we don't currently have in our Airflow docker image.
[18:01:15] <dcausse>	 yes
[18:01:24] <dcausse>	 hm...
[18:01:58] <dcausse>	 looking at how others are running hql
[18:02:31] <btullis>	 Bother. Maybe we will have to add it here? https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/blob/main/blubber.yaml?ref_type=heads#L100
[18:04:43] <dcausse>	 sigh... only search seems to be using the HiveOperator
[18:05:05] <dcausse>	 used in many places for us and might take time to migrate to something else...
[18:06:08] <btullis>	 OK. I'm in a meeting right now. Maybe we could roll back, or add the hive CLI quickly tomorrow.
[18:06:47] <dcausse>	 I see SparkSqlOperator being used
[18:21:28] <dcausse>	 but we rely on airflow templating and apparently that's not supported with this operator... :(
[18:22:48] <dcausse>	 ah no it might work if the sql is passed directly
[18:25:41] <inflatador>	 back
[18:30:14] <inflatador>	 dcausse btullis I can get started on a CR to add the hive package(s) if y'all like. Guessing it would be these packages: https://phabricator.wikimedia.org/P71951 ?
[18:30:59] <dcausse>	 inflatador: no clue what's required to just have the hive CLI
[18:31:15] <inflatador>	 it's part of the `hive` package
[18:31:20] <dcausse>	 sending a quick patch to use something else, if that works maybe we can move away from hive
[18:32:06] <inflatador>	 OK, let me know...happy to cut a new image with a few more pkgs if that's easier
[18:32:53] <btullis>	 inflatador: Let's see if the approach by dcausse works first, if we have time. I know that we prefer not to use the hive CLI, if it's possible.
[18:33:10] <dcausse>	 I think we missed that SparkSqlOperation recommendation but now I remember Joseph telling me something about this several months ago :/
[18:35:15] <inflatador>	 btullis ACK, will wait for y'all's feedback
[18:57:17] <btullis>	 dcausse: I have to step away in about 15 minutes. How's it looking from your end?
[18:57:53] <dcausse>	 btullis: well... still getting CI errors in airflow-dag :(
[18:59:11] <dcausse>	 btullis: if I don't get anywhere this evening I'll stop the dag and ponder tomorrow
[18:59:51] <btullis>	 OK. Is it just the one DAG, then? When you said 'used in many places' I thought that it might be several that were affected.
[19:01:39] <dcausse>	 it's actually 2 dags, other uses are for CREATE/ALTER statements that are not run automatically
[19:02:14] <dcausse>	 so might be doable to migrate the actual uses and do later the CREATE/ALTER ones
[19:06:32] <ottomata>	 dcausse: i think our sparksqloperator supports templating?
[19:07:01] <dcausse>	 ottomata: it does with its own param yes but wanted to avoid rewriting much of the SQL
[19:07:02] <btullis>	 Oh yes, these are your `_init` DAGs that are paused and only get run manually, right?
[19:07:09] <dcausse>	 btullis: yes exactly
[19:07:38] <dcausse>	 but if the sql is provided directly and not from a file airflow might apply templating
[19:07:47] <dcausse>	 so I think it might just work for me
[19:08:03] <dcausse>	 what's not clear is all the hints we passed to hive
[19:09:45] <dcausse>	 removed some that I believe are not required but things like "mapred.reduce.tasks: 6" to reduce to number of partitions, I'll have to understand how to do the same with sparksql 
[19:28:42] <dcausse>	 ci green, will deploy after dinner
[19:30:27] <btullis>	 Great! 
[20:11:06] <dcausse>	 bah, back ticks are removed from the query.... "csrs.`database`," -> "csrs.,"
[20:19:34] <inflatador>	 (╯°□°）╯︵ ┻━┻
[20:39:15] <inflatador>	 quick break, back in ~10
[20:39:18] <inflatador>	 err....20
[20:42:03] <dcausse>	 seems to be "spark-submit" that's subsutiting the back ticks...
[20:49:15] <dcausse>	 other projects have similar issues it seems: https://github.com/apache/incubator-livy/issues/415
[21:16:33] <dcausse>	 Fail to parse '2024-10-12 0:00:00' in the new parser. You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string.
[21:16:35] <dcausse>	 sweet
[21:16:47] <dcausse>	 ok giving up for today