[10:07:58] reminder to all to have a look at the Search SLO proposal: https://docs.google.com/document/d/1gYROXo8Fl7JSxReHAVI22EhcPvG-INVkq79a1C3tfK0/edit# [15:32:55] gehel ebernhardson meeting? [15:58:36] \o [16:38:08] given the conversation in the monthly staff meeting, I've sent an invite for anyone who needs time to debrief later today (this is conflicting with the SRE pairing session). [16:38:27] Ping me if you want a more private discussion, or if you need another time. [16:40:35] `array_position(array, element) - Returns the (1-based) index of the first element of the array as long.` gotta love things that vary between 0-based and 1-based indexing :P [17:55:24] ebernhardson: there’s only so much love you can give, I’d save it for valuable things. ;-) [17:56:45] BTW: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/249 - is there a way (by unit tests?) to see if HivePartitionRangeSensor and RangeHivePartitionSensor create similar outputs? [18:04:46] pfischer: hmm, i took a quick look over the version implemented in airflow-dags but it certainly seemed more complicated than what we were previously doing in airflow v1. It looks like it kinda folded in the ability to do things that our helper functions were doing. Lemme take a look over how we could get the results from their sensor [18:08:42] pfischer: from in the test suite you can probably do something like the following in test_dag_structure.py: https://phabricator.wikimedia.org/P44919 [18:08:57] that should, in theory, put the rendered and prepared partition names in a file [18:09:53] alternatively, there should be a way to get a REPL and manually inspect these kinds of things. We could do a call if you have some time (might take 30-60min) or i can try and work out how to do that later today and write it down somewhere [18:10:22] (the phab paste is untested, i just wrote that while glancing at the code) [18:12:06] you might have to create the appropriate directory in fixtures before running the test, i don't think it does any directory creation when missing [18:14:45] for the outputs in the old repo, i guess it would have to be slightly modified. Probably move the `if self.partition_names is None` condition into a function (pre_execute like the replacement works) and then use the for_each_task helper in a similar way [18:15:03] although note the way for_each_task is used is different in the old repo, but there are examples in the test you can follow [18:36:06] I suppose another way to get a repl, you can drop `import pdb; pdb.set_trace()` into the top of a test function, and then run pytest with the `--pdb` argument and you will be dropped into a debugger inside that function [18:36:20] it's a gdb-ish cli debugger [18:37:25] without the set_trace call it will drop you into a debugger when an uncaught exception is thrown from a test [18:44:03] pfischer: i suppose i couldn't help poking it a bit...here is what worked for me: https://phabricator.wikimedia.org/P44920 [18:45:18] i'm left to wonder if this will work for all our use cases though, we dont always want year/month/day/hour partitioning, sometimes we want a range of date= partitions. Can figure that out when we get there though [18:51:46] [pro-tip] from Python 3.7 you can use breakpoint() anywhere in the code ;) https://docs.python.org/3/library/functions.html#breakpoint [18:52:26] volans: oh cool, i guess i'm ancient still importing pdb to do the same thing :) [18:52:35] 3.7 isn't exactly new [18:52:45] not bleeding edge, no :-P [19:18:40]