[08:45:18] <pfischer>	 o/ dcausse: Good morning! Are you around for another spark 3 deployment/DAG update round? Would 10:00 suit you?
[08:46:41] <dcausse>	 pfischer: hey! I might be distracted by the k8s upgrade in codfw but we can try 
[08:47:23] <dcausse>	 could also be something interesting for you to follow
[09:01:15] <pfischer>	 Sure, I’m happy to do so.
[09:01:26] <dcausse>	 pfischer: I'll be in https://meet.google.com/fad-kypz-inn for the next hour, feel free to join
[09:08:30] <gehel>	 ryankemper: is there something more on T325324 ? Or should we move it to needs reporting?
[09:08:30] <stashbot>	 T325324: Evaluate options to soften wdqs paging - https://phabricator.wikimedia.org/T325324
[10:01:58] <gehel>	 dcausse: do you want to cancel our 1:1 due to the k8s upgrade?
[10:02:10] <dcausse>	 gehel: oh yes sorry about that
[10:02:22] <gehel>	 no problem, that sounds like the right priority!
[10:48:04] <gehel>	 lunch
[10:58:06] <dcausse>	 lunch
[11:16:12] <pfischer>	 lunch
[14:06:46] <inflatador>	 <o/
[14:07:17] <dcausse>	 o/
[14:09:58] <inflatador>	 gehel thanks for your help with the switch maint, sorry I missed it!
[14:42:29] <gehel>	 inflatador: no problem!
[14:43:13] <gehel>	 note that I did not ban the nodes from the cluster, given the short timeframe. And that I was fairly confident that this would work just fine without banning them
[14:43:43] <gehel>	 inflatador: can you repool all servers? Looks like the maintenance is ove
[14:43:49] <inflatador>	 yeah, that's what we did for the last switch maintenance. I forgot about this one though ;(
[14:43:50] <gehel>	 s/ove/over/
[14:43:55] <inflatador>	 ACK, will do
[14:44:59] <gehel>	 inflatador: note that we depooled the whole wdqs service from codfw, as the updater was down for the k8s upgrade
[14:45:29] <gehel>	 repooling the individual wdqs nodes should be fine, as long the the service is still depooled (and depooling the nodes was probably not necessary)
[14:45:42] <gehel>	 dcausse might have more context on the wdqs side
[14:46:13] <gehel>	 and now that I think about it, did we also depool wcqs? It should be affected in the same way for the updater upgrade?
[14:46:14] <dcausse>	 we're waiting for the k8s upgrade to be done before restarting the rdf-streaming-updater job
[14:46:20] <inflatador>	 Cool, I'll work on Elastic for now
[14:46:33] <gehel>	 quick errand (need to get food for this evening)
[14:46:44] <gehel>	 dcausse: can we do our 1:1 in ~20' when I'm back?
[14:46:54] <dcausse>	 sure
[14:46:59] <gehel>	 I'll ping you
[14:47:05] <gehel>	 see you!
[14:47:10] <dcausse>	 see you!
[14:47:29] <inflatador>	 dcausse that might affect our meeting but if k8s is being updated, maybe we can't do anything. Anyway we can always meet tomorrow if need be
[14:48:21] <dcausse>	 oh right :/
[14:48:47] <dcausse>	 I think we can still meet if you have time
[14:49:06] <dcausse>	 we'll have to restart the rdf-streaming-updater in codfw anyways today
[14:52:08] <inflatador>	 OK, whenever you're done with g-ehel hit me up
[14:52:58] <dcausse>	 sure we can quickly meet before as well if you have time (to sync-up on the wdqs reload and k8s@codfw)
[14:53:26] <inflatador>	 dcausse sure, I'm up at https://meet.google.com/qve-fycn-vpw now
[14:53:36] <dcausse>	 cool, joining
[15:12:05] <gehel>	 dcausse: I'm in https://meet.google.com/meg-wrep-dru for when your done with Brian
[15:27:28] <dcausse>	 inflatador: I joined the same meet to do the rdf-streaming-updater restart if you want
[15:58:22] <ebernhardson>	 \o
[15:58:30] <inflatador>	 <o/
[15:59:12] <dcausse>	 o/
[16:21:39] <ebernhardson>	 looks like about 6 failures reindexing across all clusters, not that bad. Mostly on cloudelastic. Started up a few, will start up the rest once some of these have finished
[16:46:44] <pfischer>	 o/ ebernhardson: I’m migrating import_ttl.py to analytics-dags but my IDE still complains about missing imports (the build/test is running, however). What setup do you use for editing those dags?
[16:47:37] <ebernhardson>	 pfischer: for editing python code i typically use vim, to check for missing imports i typically run the pytest suite
[16:48:18] <ebernhardson>	 mypy also does the static analysis bit, i suppose it would complain about missing imports as well (pytest and mypy should both be triggered by tox)
[16:48:47] <gehel>	 Dinner, back later 
[16:55:45] <ebernhardson>	 pfischer: which deps is it not finding? It might be that we are depending on something thats default-installed to debian instances but not listed in the airflow/setup.py 
[17:00:26] <inflatador>	  wdqs/wdqs-internal/wcqs CODFW are now repooled
[17:01:00] <dcausse>	 ebernhardson: do you use miniconda?
[17:01:24] <ebernhardson>	 dcausse: hmm, not sure. For the new repo i use the blubber image 
[17:02:14] <ebernhardson>	 it installs the conda-analytics package from wmf which has a variety of things i didn't look into directly
[17:02:15] <dcausse>	 oh ok and you run tox from that image?
[17:02:17] <ebernhardson>	 ya
[17:03:12] <ebernhardson>	 or i just run the pytest/flake8/etc. commands directly in the image, since the tox configuration here isn't setup to allow running specific things and runs it all
[17:03:56] <ebernhardson>	 oh nevermind, thats the airflow-dags repo that has them all bundled together. the discolytics can use `tox -e flake8` or whatever
[17:05:20] <dcausse>	 I think we're trying to setup this in intellij and thus having all the deps in a conda env
[17:05:29] <dcausse>	 that intellij can point to
[17:07:46] <inflatador>	 workout, back in ~40
[17:08:54] <ebernhardson>	 dcausse: for discolytics it should have most of them, but it's possible we are depending on something that the conda-analytics package provided. The package is almost a GB so seems plausible something in there that we need to add to our conda-environment.yaml to be explicit
[17:10:19] <ebernhardson>	 can get a package list from /opt/conda-analytics/conda-meta/*.json on one of the stat hosts i think
[17:10:34] <dcausse>	 that's where I'm a struggling a bit
[17:15:24] <dcausse>	 for instance I don't get why tox is running fine but the conda env it's running in does not have the deps in the end
[17:15:59] <dcausse>	 ah tox installs stuff in .tox/py37/lib/python3.7/site-packages/ hm...
[17:16:50] <ebernhardson>	 hmm, tox also installs requirements-test.txt, those are test-only requirements like pytest and mypy
[17:22:46] <dcausse>	 python's so mysterious to me
[17:23:33] <dcausse>	 finally ran "pip install ." in the right conda env (found that in "run_dev_instance.sh")
[17:23:49] <dcausse>	 and intellij is now happy
[17:23:52] <ebernhardson>	 python packaging is certainly a mess
[17:25:29] <dcausse>	 I guess it's sourcing the package from setup.cfg
[17:27:13] <ebernhardson>	 toying with intellij, it doesn't seem like adding a conda environment actually imported the conda-environment.yaml
[17:27:27] <ebernhardson>	 i suppose the pip install . might have done that for you
[17:27:32] <dcausse>	 where did you find that conda-env.yaml?
[17:29:34] <ebernhardson>	 i'm mucking about in discolytics.  The airflow-dags repo probably depends on the conda environment installed by the airflow package andrew made, sec
[17:30:28] <dcausse>	 oh sorry was talking about  airflow-dags
[17:31:17] <ebernhardson>	 see https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/+/refs/heads/debian
[17:31:27] <dcausse>	 looks like they're adding such yaml file in https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/200
[17:32:27] <ebernhardson>	 setup.cfg in the airflow-dags repo might cover it all though, since it manages to run in the tests
[17:33:11] <dcausse>	 ok
[17:34:04] <dcausse>	 making a quick mr to see if I'm having this right
[17:34:47] <dcausse>	 switching from airflow.operators.dummy_operator to airflow.operators.dummy (former is deprecated)
[17:35:00] <ebernhardson>	 people love renaming and moving things around :)
[17:35:33] <dcausse>	 :)
[17:36:30] <ebernhardson>	 i guess it's nice execution_date got renamed to data_interval_start though, less foot-gun'y
[17:37:38] <dcausse>	 oh indeed
[17:47:36] <inflatador>	 dcausse for the data transfer, the arg '--blazegraph_instance' should be 'wikidata' for WDQS? Do we need to do a separate transfer for 'categories'?
[17:48:23] <dcausse>	 inflatador: I need to re-read the code can't remember
[17:48:41] <inflatador>	 https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-transfer.py
[17:48:49] <dcausse>	 (we don't want to transfer the categories journal)
[17:49:53] <dcausse>	 inflatador: yes --blazegraph_instance wikidata should be what we need
[17:50:06] <dcausse>	 you can test transfering to wdqs1009 first
[17:50:18] <inflatador>	 dcausse ACK, starting first xfer (from 1010 to 1009) now
[17:50:23] <dcausse>	 (stopping the current reload if it's still running)
[17:52:16] <inflatador>	 dcausse oh crap, I reversed the source and dest. I stopped myself in time though ;)
[17:52:31] <dcausse>	 ouch :)
[17:53:11] <inflatador>	 OK, here goes
[17:56:40] <inflatador>	 instant failure...looks like encrypt/decrypt is the problem again
[17:57:16] <dcausse>	 :(
[17:59:27] <inflatador>	 we'll probably have to hack around that...if anyone knows of a python library that would handle that transparently LMK. In the meantime I'll remove the encryption part from the cookbook and give a heads-up when it's back
[17:59:34] <inflatador>	 I mean, when it's ready to run again
[18:00:35] <dcausse>	 should we run a checksum at the end to make sure that the data is sane?
[18:00:48] <dcausse>	 (if we remove encryption)
[18:03:33] <inflatador>	 we're doing some rudimentary sanity checking here https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-transfer.py#L123
[18:03:44] <inflatador>	 but yeah, checksum could be good
[18:04:08] <inflatador>	 shasumming a 1.5T file might take awhiel, if there is a more efficient way to do that LMK
[18:04:15] <dcausse>	 do you have a particular error on the instant failure btw?
[18:04:51] <inflatador>	 bad decrypt
[18:04:51] <inflatador>	 140321574442176:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:../crypto/evp/evp_enc.c:610:
[18:04:55] <dcausse>	 this transfer technique used to work a year ago, I wonder what broke it
[18:05:32] <inflatador>	 It wasn't reliable with encryption, probably due to network issues outside of our control. We tried to simplify the encryption command but I'm not sure we helped much ;(
[18:06:40] <inflatador>	 theoretically, we could copy the files to the NFS server and have the rest of the hosts pull from there, but we've been asked not to use NFS if possible
[18:07:39] <dcausse>	 hm... I think we want the source to catch up on lag before transfering to another host (so that the lag does not cumulate)
[18:08:50] <inflatador>	 ACK. Other SREs have also asked for a helper function or library for long-running transfers so I might check in with v-olans on that
[18:12:47] <inflatador>	 anyway, probably won't start anything for awhile
[18:13:17] <inflatador>	 ryankemper gehel  I won't make pairing today. I have to take my son to the doctor for a couple hours. I'll be online an working from there though
[18:13:35] <ebernhardson>	 hmm, isn't super clear how to run refinery-drop-older-than from airflow 2.  they didn't package up refinery/python as an installable package, so we can't really bundle it into discolytics without forking
[18:13:52] <inflatador>	 ryankemper we can probably touch base once I get back (~2 pm your time) and work on the transfer cookbook
[18:14:33] <ebernhardson>	 tempted to fork the relevant bits, a bit messy though and puts us on the hook for future maintenance
[18:14:34] <inflatador>	 ebernhardson are you using an-airflow1005 yet? I was under the impression we had to change some hieradata stuff so you could have a postgres DB, but havent' checked in yet this wk
[18:14:58] <ebernhardson>	 inflatador: nope, it's not usable yet afaik
[18:16:30] <inflatador>	 ebernhardson ACK, that confirms my assumption. I'll try and look at that today as well
[18:17:11] <inflatador>	 quick lunch, back in ~1h or so
[19:05:22] <inflatador>	 back 
[20:30:38] <inflatador>	 re: an-airflow1005 . Still confirming w/Andrew and Ben, but my current guess is that we would update https://gerrit.wikimedia.org/r/c/operations/puppet/+/883680/4/hieradata/role/common/analytics_cluster/airflow/search.yaml#45 to point to the new postgres DB shown in https://gerrit.wikimedia.org/r/c/operations/puppet/+/889572/2/hieradata/role/common/analytics_cluster/postgresql.yaml#2
[20:41:25] <ryankemper>	 gehel: re T325324, I need to look into whether it's feasible to tune the pybal-related alerts as well. after that the ticket will be done
[20:41:26] <stashbot>	 T325324: Evaluate options to soften wdqs paging - https://phabricator.wikimedia.org/T325324
[20:43:52] <ebernhardson>	 inflatador: next up then would be figuring out why the scap deployment of the dags failed,  /srv/deployment/airflow-dags/search is missing on an-airflow1005 but defined in puppet. iirc we had some issue with a puppet run previously related to that
[20:44:08] <ebernhardson>	 i suppose i can try a deploy from deployment host and see if it ships
[20:45:12] <inflatador>	 ebernhardson cool, let me know how it goes. I'll be back home in 30m and can dig deeper if that doesn't do the trick
[20:45:13] <ebernhardson>	 from deployment host: sign_and_send_pubkey: signing failed: agent refused operation
[20:45:15] <ebernhardson>	 analytics-search@an-airflow1005.eqiad.wmnet: Permission denied (publickey,keyboard-interactive)
[20:45:26] <inflatador>	 Ah, OK
[20:46:18] <inflatador>	 I have no idea how scap works, but it sounds like an SSH problem?
[20:47:02] <ebernhardson>	 something like that. What should be happening here is it ssh's from the deploy host into the an-airflow1005 instance and runs a scap command there (that then pulls from the deployment host)
[20:48:43] <ebernhardson>	 maybe my scap config is wrong there, or some other puppet config is necessary. In the past we used `deploy-service` as the ssh_user, here we use `analytics-search`.   The other airflow-dags deployments also use the destination user but maybe they have additional configuration
[20:49:51] <inflatador>	 yeah, seeing the same thing in /etc/passwd
[20:50:23] <inflatador>	  /srv/deployment/airflow-dags/search exists on 1005 FWiW 
[20:50:59] <ebernhardson>	 oh indeed, it looks like even though scap bailed it managed to put some files in place
[20:52:57] <inflatador>	 If I manually run the same command (without sudo) as the error code (`/usr/bin/scap deploy-local --repo airflow-dags/search -D log_json:False`) ...
[20:54:39] <inflatador>	  ... I get an error `fatal: detected dubious ownership in repository` 
[20:54:49] <inflatador>	 not sure if that actually tells us anything
[20:55:28] <ebernhardson>	 hmm, not sure either
[20:56:41] <ebernhardson>	 it probably wants to be run as the analytics-search user
[20:58:14] <ebernhardson>	 probably unrelated, but i have NOPASSWD: ALL for analytics-search sudo rights on an-airflow1001 but not 1005
[20:58:36] <ebernhardson>	 (and on stat100* hosts)
[21:03:24] <ebernhardson>	 and looks like the airflow webserver is trying to start, but is in a bootloop
[21:04:03] <ebernhardson>	 ERROR - DB Creation and initialization failed: (MySQLdb._exceptions.OperationalError) (1045, "Access denied for user 'airflow_search'@'2620:0:861:106:10:64:36:11' (using password: YES)")
[21:04:14] <ebernhardson>	 suggests it's not pointed at postgresql
[21:12:16] <inflatador>	 back
[22:41:42] <inflatador>	 ryankemper gave the data transfer c-ookbook a little push, and it looks like it's working this time. ~300G/1.2T copied so far