[07:11:07] <dcausse>	 regarding this BC cross cluster setting, I did play with them a bit, took the binary file holding those on the master node and checked what will happen when we migrate with es7, it properly removed them
[07:11:40] <dcausse>	 but yes... that's horrifying that we can't do anything about them while running 6
[08:42:00] <tanny411>	 dcausse: Do I have to look anywhere else other than the airflow folder in analytics to make changes related to the refactoring?
[08:42:45] <dcausse>	 tanny411: no it should be only the spark submit hooks there that need to be changed I think
[08:42:59] <tanny411>	 Okay, great!
[10:16:35] <dcausse>	 lunch
[11:44:40] <kostajh>	 did something change recently with UpdateSearchConfig.php ? When I try to run it, I get this error `ERROR: Param indexSuffix is required`
[12:10:09] <dcausse>	 kostajh: hm... yes it's --indexSuffix instead of --indexType but we added some back-compat support IIRC, some somehing not working as expected
[12:14:08] <kostajh>	 dcausse: what should the indexSuffix/Type be? I don't understand the difference between 'general' and 'content'
[12:15:14] <dcausse>	 kostajh: oh you mean that you don't even use "--indexType" when running UpdateSearchConfig.php?
[12:15:22] <kostajh>	 I'm also struggling to figure out why the jobs that are enqueued and executed via runJobs.php don't result in the search index being updated
[12:15:58] <kostajh>	 dcausse: I'm mostly following this guide (https://www.mediawiki.org/wiki/MediaWiki-Docker/Extension/CirrusSearch), are there other more official notes on how to set up indexing?
[12:16:21] <dcausse>	 if UpdateSearchConfig.php fails it's bad
[12:16:28] <dcausse>	 lemme check
[12:17:39] <kostajh>	 hmm, mysteriously, `php extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --startOver` no works. One thing I fixed in the last few minutes was allocating more disk space to Docker, because I was seeing "flood stage watermark" messages in the logs
[12:17:50] <kostajh>	 so I guess that might have impacted what was going on, I don't know for sure.
[12:18:43] <dcausse>	 kostajh: do you run elasticsearch 6.8.23?
[12:19:22] <kostajh>	 dcausse: yes, but a custom built version, because I'm on a laptop with an Apple M1 chip
[12:19:22] <stashbot>	 M1: MediaWiki Userpage - https://phabricator.wikimedia.org/M1
[12:19:34] <kostajh>	 https://gitlab.wikimedia.org/kharlan/wmf-elasticsearch-arm64/
[12:20:12] <dcausse>	 kostajh: do you still have the cmdline and the output of UpdateSearchConfig.php failing with "ERROR: Param indexSuffix is required" ? 
[12:20:23] <kostajh>	 it was working ~1 month ago, but for some reason no index contents are updated anymore.
[12:21:38] <kostajh>	 https://www.irccloud.com/pastebin/itrWDJRe/
[12:21:45] <kostajh>	 dcausse: ^
[12:22:27] <dcausse>	 thanks
[12:24:46] <dcausse>	 kostajh: regarding jobs do you see anything particular in: docker-compose logs mediawiki-jobrunner
[12:25:43] <kostajh>	 Despite linking to MediaWiki-Docker's help page, I'm not using it, sorry for the confusion.
[12:25:54] <kostajh>	 My local site is running with homebrew packages, and I use Docker just for ElasticSearch
[12:28:28] <dcausse>	 can't understand how this "ERROR: Param indexSuffix is required" could have happened :(
[12:28:55] <kostajh>	 me neither, but since I don't see it anymore, I suppose we could ignore it for now?
[12:29:10] <kostajh>	 I see an index set up when I inspect localhost:9200
[12:29:15] <kostajh>	 but no content is added to it
[12:29:25] <dcausse>	 regarding pages not getting indexed my best bet would be to inspect jobrunner logs
[12:29:43] <dcausse>	 running extensions/CirrusSearch/maintenance/ForceSearchIndex.php should force some of these jobs to be created
[12:39:34] <kostajh>	 ugh, found it. I had been working on some PHPUnit changes. As part of that I had enabled Wikibase as it is run in CI, which includes Wikibase.ci.php. And that sets... `$wgCirrusSearchDisableUpdate = true;`. Stepping through the debugger eventually got me there. it would be nice if somehow we could output a warning in the logs or stdout when CirrusSearchDisableUpdate or DisableSearchUpdate are true
[13:03:00] <inflatador>	 greetings
[13:07:26] <dcausse>	 o/
[13:07:36] <inflatador>	 <o/
[13:08:16] <inflatador>	 I keep seeing notifications that  wdqs1016.eqiad.wmnet is being reimaged in the operations channel, any idea what's up with that?
[13:09:31] <dcausse>	 inflatador: no clue what's going on but it should be tracked in T307138 
[13:09:31] <stashbot>	 T307138: Q4:(Need By: TBD) rack/setup/install wdqs101[4,5,6] - https://phabricator.wikimedia.org/T307138
[13:11:15] <inflatador>	 Thanks dcausse , looks like it's been failing for some time, but nothing we can really do except wait for DC Ops
[13:11:59] <dcausse>	 yes, Chris is taking care of it I think
[13:24:46] <ejoseph>	 errand
[13:43:45] <inflatador>	 dropping off kids, back in ~20
[14:02:49] <inflatador>	 back
[15:14:29] <ebernhardson>	 \o
[15:18:49] <dcausse>	 o/
[16:13:31] <inflatador>	 workout/physical therapy, back in ~45
[16:49:07] <inflatador>	 back
[17:22:20] <inflatador>	 lunch, back in ~1 hr
[17:37:41] <ebernhardson>	 todays just a bad day...going to take the rest off
[18:34:09] <inflatador>	 back
[18:34:26] <inflatador>	 Hope ya feel better soon e-bernhardson