[07:38:06] <dcausse>	 re: redirects left in old targets, I think it happens with the current pipeline as well. I believe that the RedirectAndIncomingLinksJob that we moved to an offline batch job used to take care of this
[11:02:26] <dcausse>	 lunch
[14:19:53] <inflatador>	 <o/
[15:57:08] <Trey314159>	 P&T staff meeting or Wednesday meeting in 3 minutes?
[15:59:35] <inflatador>	 Trey314159 wondering the same thing ;)
[16:00:47] <Trey314159>	 I guess we can talk about it in the Wednesday meeting, then!
[16:18:20] <pfischer>	 ebernhardson: could you narrow down the JSON serialisation issue in the consumer? I’ve been running the local setup all day and didn’t run into any such issue.
[16:21:42] <ebernhardson>	 pfischer: sorry, i pushed a patch yesterday but forgot to create the MR: https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/57
[16:32:49] <pfischer>	 ebernhardson: good catch, thank you! BTW: did you adjust the elastic emitter bulk settings (elasticsearch-bulk-flush-interval, elasticsearch-bulk-flush-max-actions, elasticsearch-bulk-flush-max-size-mb)?
[16:34:38] <pfischer>	 If we have an SLO for the max. acceptable lag, then we should at least cap the interval.
[16:34:53] <ebernhardson>	 pfischer: no i don't think we set any of those yet
[17:14:24] <inflatador>	 workout, back in ~40
[17:46:00] <inflatador>	 back
[17:52:44] <inflatador>	 wdqs1024.eqiad.wmnet failed its data-reload again, starting over
[18:00:48] <inflatador>	 Also, rolling restart of relforge happening now
[18:05:58] <inflatador>	 relforge done, starting prod eqiad
[18:43:37] <inflatador>	 lunch, back in ~45
[19:35:39] <inflatador>	 back
[19:53:03] <inflatador>	 Created a ticket for the backfill test stuff we talking about in Weds mtg: https://phabricator.wikimedia.org/T350826 . Apologies in advance if my notes suck
[20:04:14] <ebernhardson>	 hmm, so the problem with fixing the redirect in the producer is that we still have invalid records in the update topic to be read by the consumer
[20:07:31] <ebernhardson>	 i suppose i can simply bypass those for this time around, but maybe we need a plan in the future when it's not so easy to skip over inputs
[20:23:11] <ebernhardson>	 except, hmm...I can move the consumer group position forward by an arbitrary number of records with kafkacat as long as flink isn't also subscribed to the topics, but then how do i convince flink to start consuming again.
[20:24:17] <ebernhardson>	 and producer fell over now:P Caused by: java.lang.NoSuchMethodError: 'org.slf4j.spi.LoggingEventBuilder org.slf4j.Logger.atDebug()'
[20:24:26] <ebernhardson>	 seems perhaps a version conflict, or something we marked provided that isn't
[21:34:44] <inflatador>	 not feeling great, going to finish a little early today
[22:50:38] <moritzm>	 tomorrow when I roll out Java 8 updates, should I extend https://phabricator.wikimedia.org/T350703 with the Blazegraph or do you prefer a new/separate task (given these use Java 8)=
[22:50:40] <moritzm>	 tomorrow when I roll out Java 8 updates, should I extend https://phabricator.wikimedia.org/T350703 with the Blazegraph or do you prefer a new/separate task (given these use Java 8)?
[22:52:34] <ryankemper>	 moritzm: extending the existing ticket is fine, thanks
[22:55:28] <moritzm>	 ack, will do