[08:28:09] thanks for fixing all the dags... must have been tedious [08:28:43] fixing export_queries_to_relforge, for some reason this one runs on minute "38" rather than "00" :/ [09:29:41] errand, might come back only after lunch [12:31:04] Errand+lunch [14:10:09] for awareness, there are a couple of scap issues causing widespread puppet failures ATM https://phabricator.wikimedia.org/T326668#8640584 [14:11:33] one of them is affected the new airflow VM [14:11:40] affectING that is [14:27:34] Hey y'all, I'm preparing the patches for the datacenter switchover, is this https://wikitech.wikimedia.org/wiki/Switch_Datacenter#ElasticSearch still needed (the more_like hardcode) ? [14:48:20] claime: now that we are running active-active, the warm up should not be needed anymore. [14:48:41] ebernhardson, inflatador: could you review the documentation above and see what needs to be updated? [14:48:51] Thanks <3 [15:00:59] dcausse will be about ~5m late [15:01:23] np! [16:02:25] errand, back in ~45 [16:02:56] mpham, ebernhardson: retrospective: https://meet.google.com/eki-rafx-cxi [16:44:24] ebernhardson: Last night I couldn’t make it. Would you have some time to walk me through deploying import_ttl in 15min? [16:44:31] pfischer: sure [16:44:45] Awesome, I’ll send an invite. [16:50:47] back [16:51:06] gehel ACK will check docs [17:32:19] ebernhardson or anyone else, can you confirm that the DC switch docs on our own page have all the steps? https://wikitech.wikimedia.org/wiki/Search#DC_Switch . I'll update the Switch Datacenter page accordingly once confirmed [17:39:08] inflatador: looks good to me, I think we can remove the note regarding "Having search traffic flow between 2 datacenters increases the privacy risks" now [17:42:42] dcausse ACK thanks, will get rid of the sections on preparation in https://wikitech.wikimedia.org/wiki/Switch_Datacenter#ElasticSearch then? [17:42:50] Also heading to lunch, back in ~1h [17:43:08] there might be better APIs to see the actual conf, looking [17:44:44] inflatador: oops, updating the link to the config, cirrus conf has been moved [17:55:57] inflatador: updated the doc [18:31:53] Back [18:32:02] and thanks dcausse ! [19:11:10] dinner [19:37:10] ebernhardson: we removed the whole section about cache warm up: https://wikitech.wikimedia.org/w/index.php?title=Switch_Datacenter&diff=2056089&oldid=2055896 could you have a look and confirm? [19:37:29] My understanding is that we don't need to do anything for Search now that we are active-active [19:47:39] ebernhardson: if you're ok with the doc, just move T330417 to "needs reporting" [19:47:40] T330417: Update Elasticsearch documentation around datacenter switches - https://phabricator.wikimedia.org/T330417 [19:48:30] done [19:51:15] ebernhardson: thanks! [20:24:06] ebernhardson: BTW: import_ttl finished [20:24:21] pfischer: nice! [20:24:58] i took a slightly different approach to fixing the pytest problems, i've instead worked out a way that will let us parametrize tests on the individual task instances. It's mostly working but i'm still working through a few things [20:25:17] that way we go back to one failure per test, more like how tests are typically run [20:30:33] at least i hope i have...the current error is somehow SparkSubmitOperator thinks we are using master=local, when it should have seen master=yarn and i haven't yet understood why [20:32:21] suggests some order-of-operations problem with environment variables perhaps [20:37:26] Alright, thank you. I’m looking forward to your fix! [20:50:29] break, back in ~30 [22:09:43] ryankemper elastic restarts are finished in codfw prod. You can start up eqiad if you want or I'll kick it off tomorrow. I already updated the ticket w/progress [22:27:24] afk, school run [22:34:26] inflatador: excellent, kicking it off now [22:36:20] ACK, thanks [22:55:32] back