[08:47:13] <gehel>	 Errand 
[09:35:44] <pfischer>	 o/ dcausse: would you have some time for looking at the cirrus-search-update flink-application deployment with me? I’m running into timeouts for kafka batch creation despite passing config to disable batching.
[09:35:59] <dcausse>	 pfischer: sure!
[09:38:39] <pfischer>	 https://meet.google.com/smb-vnfb-pjv?authuser=0
[11:12:33] <dcausse>	 lunch
[14:07:23] <inflatador>	 <o/
[14:10:48] <dcausse>	 o/
[14:27:46] <inflatador>	 dcausse had a question about wdqs data reload. We ran it on wdqs2009 and it got all the way to the end, but then erred with message "--kafka-timestamp should be set when reloading commons or wikidata" . Do we need to start it again? I vaguely remember setting that timestamp value around the time of the dump?
[14:28:30] <dcausse>	 did you reload from dc to another (eqiad -> codfw) ?
[14:29:18] <inflatador>	 no, , we got it from that https://dumps.wikimedia.your.org site
[14:29:37] <dcausse>	 oh you mean a full reload?
[14:29:49] <dcausse>	 sorry  (tought it was a transfer)
[14:29:53] <dcausse>	 hm..
[14:29:59] <inflatador>	 yeah, per https://phabricator.wikimedia.org/T323096
[14:30:42] <dcausse>	 a wikidata reload takes several days (8+) when did you started it?
[14:31:20] <inflatador>	 last week
[14:31:53] <inflatador>	 Thursday, I think? Let me check. But it was showing that error message when I looked at it on Monday. I'll paste the full error message, 1 sec
[14:32:26] <dcausse>	 seems to short, looking at dates I doubt it has done anything
[14:32:31] <dcausse>	 s/to/too
[14:33:09] <dcausse>	 dumps are on 8 dec but the journal is still at dec 7
[14:34:15] <inflatador>	 Sounds like it didn't work then. The error also indicates it only ran for 8 hrs: https://phabricator.wikimedia.org/P42685
[14:34:45] <dcausse>	 yes it definitely did not work
[14:35:27] <dcausse>	 getting the right timestamp is not entirely trivial on wikidata because we have 2 dump files
[14:35:28] <inflatador>	 OK, I will start it again. What should I set for kafka timestamp?
[14:35:32] <inflatador>	 ah
[14:40:12] <dcausse>	 bzcat $DUMP_FILE | head -50 | grep wikibase:Dump -A 5 | grep schema:dateModified | sed -e 's/.*"\(.*\)".*/\1/'
[14:40:29] <dcausse>	 is how to extract the timestamp out of a ttl dump
[14:40:47] <dcausse>	 then the oldest timestamp out of the two dumps must be taken
[14:41:38] <dcausse>	 if the timestamp is older than the kafka retention period + the estimated time to do the import then the import should not be made
[14:41:58] <inflatador>	 estimated time is ~8 days you said?
[14:42:34] <dcausse>	 yes more or less, but since data size is only growing it's hard to estimate
[14:42:36] <ottomata>	 btw, you (maybe) can have more retention if you need, which topic is this?
[14:43:10] <dcausse>	 ottomata: it's the wdqs mutation topic (lemme find the actual name)
[14:44:33] <inflatador>	 hmmm, bzcat cmd returns '2022-08-22T23:00:10Z' as the date. That seems way too old for a dump we downloaded last week, let me check on that
[14:44:50] <ottomata>	 eqiad.rdf-streaming-updater.mutation ?
[14:45:25] <dcausse>	 yes but here it'll backfill from codfw.rdf-streaming-updater.mutation on kafka-main@codfw
[14:45:52] <inflatador>	 ah, sure enough those files seem way out of date at https://dumps.wikimedia.your.org/other/wikibase/wikidatawiki/
[14:46:29] <inflatador>	 sounds like we probably need to revisit https://gerrit.wikimedia.org/r/c/operations/puppet/+/832543/ and mount the dumps via NFS
[14:47:01] <dcausse>	 this mirror seems out of date indeed :(
[14:48:37] <inflatador>	 NP, I'll add wdqs2009 to that patch and have it up again shortly
[14:48:58] <inflatador>	 may also have to update the cookbook
[14:49:26] <dcausse>	 yes it'd be great if the cookbook could autodetect the kafka timestamp to use
[14:50:38] <ottomata>	 ya those topics don't look that huge, we could probably double the retention if that would be helpful?
[14:51:43] <dcausse>	 might be helpful indeed
[14:53:35] <dcausse>	 retention seems to be around 1month at the moment
[14:55:21] <pfischer>	 dcausse: FYI kafka-test brokers only support produce API level 5, zstd compression is supported from 7 on.
[14:55:44] <dcausse>	 pfischer: oh interesting did not know that
[14:55:47] <dcausse>	 ottomata: ^
[14:55:55] <ottomata>	 OH its already more than 7 days, nice.  yeah sure, dcausse file a task if you decide you want it :)
[14:56:16] <ottomata>	 oh pfischer interesting.  welp finally a reason to upgrade kafka
[14:56:45] <dcausse>	 we're going to force snappy for simplicity reasons anyways so no big deal but good to know
[15:17:50] <inflatador>	 OK, NFS patch is ready if anyone can take a look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/867646 
[15:43:16] <dcausse>	 errand
[15:57:13] <inflatador>	 thanks for the review! still failing PCC but I think it's because we listed a deprecated host (labstore1007.wikimedia.org) in the commit msg. Pinged in wikimedia-cloud to take a look
[17:01:25] <inflatador>	 workout, back in ~40
[19:08:18] <dcausse>	 dinner
[20:16:13] <inflatador>	 OK, finally back from lunch & training
[21:08:43] <Trey314159>	 ryankemper: I saw that the Ukrainian plugin has been deployed! Thanks!
[21:09:07] <ryankemper>	 np!
[22:58:46] <inflatador>	 OK, NFS is mounted on the W[CD]QS servers, more details in https://phabricator.wikimedia.org/T323096