[10:31:20] <dcausse>	 errand+lunch
[11:04:44] <gehel>	 dcausse, dr0ptp4kt: I can't make it to the Graph Split log analysis meeting this afternoon. I'll read the notes and I'm sure you can move forward without me!
[11:32:40] <dr0ptp4kt>	 👍 
[13:42:53] <gehel>	 dcausse, inflatador : we have a thanisnspace usage alert for the WDQS streaming updater that seems to be open since quite a long time. Should we do something about it!
[13:43:26] <gehel>	 See #wikimedia-operations : 2:38 PM <• jinxer-wm> (RdfStreamingUpdaterSpaceUsageTooHigh) firing: (2) The RDF Streaming Updater is using more than 50GiB of storage - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater - https://grafana.wikimedia.org/d/fdU5Zx-Mk/wdqs-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DRdfStreamingUpdaterSpaceUsageTooHigh
[13:43:48] <dcausse>	 gehel: yes, we should definitely clean this up and start learning how we could automate this process
[13:44:32] <gehel>	 Ok, I'll create a ticket
[13:45:12] <dcausse>	 thanks!
[13:57:59] <dcausse>	 ryankemper: per yesterday's discussion I filed T355888
[13:57:59] <stashbot>	 T355888: Enable cross federation between experimental WDQS endpoints - https://phabricator.wikimedia.org/T355888
[14:12:33] <inflatador>	 <o/
[14:12:55] <inflatador>	 gehel Y I just saw that one yesterday...perfect example of an alert that should create a ticket. Anyway, I'll take a look
[14:14:35] <dcausse>	 o/
[14:38:34] <moritzm>	 inflatador: shall I go ahead and switch wdqs::internal to Puppet 7 next? or do you prefer to first go with a single canary for that role?
[15:05:27] <inflatador>	 moritzm Y, sorry I haven't been able to move that fwd. Feel free to migrate wdqs:internal to Puppet 7
[15:08:36] <moritzm>	 ok, I'll do that now, then
[15:21:12] <moritzm>	 inflatador: wdqs/internal is done, I'll also proceed with the wcqs role, ok? the canary host (wcqs2001) is on Puppet 7 since Nov 22
[15:24:57] <inflatador>	 moritzm excellent, feel free to proceed
[15:25:12] <moritzm>	 ack, on it
[15:35:05] <moritzm>	 wcqs is now also done, I'll do wdqs::public on Monday, the switch maintenance is upcoming and I don't want to run the migraton (which takes a bit for 24 hosts) while it lasts
[15:39:43] <inflatador>	 moritzm understood. Feel free to give me a heads-up if you into issues. So far it's been pretty seamless (at least for us, you might have a different take ;) )
[15:41:03] <moritzm>	 yeah, I wouldn't expect any major issues for search related systems at this point
[15:41:39] <moritzm>	 for the cirrus main cluster it would be good to switch a few more canaries before the migrate the entire role, let's catch up on Monday for that
[15:58:57] <Trey314159>	 inflatador: If you can get The Wordhord from your library, then it's worth checking out (in both senses!)
[16:02:56] <ebernhardson>	 \o
[16:11:48] <inflatador>	 <o/
[16:17:55] <ebernhardson>	 hmm, patch 992974...is it time to start planning on how to capture patch 1000000? :)
[16:30:44] <dcausse>	 o/
[16:30:46] <dcausse>	 :)
[16:39:30] <ebernhardson>	 hmm, i think ready to turn on SUP in prod..wondering if i'm forgetting anything
[16:43:59] <dcausse>	 sorry haven't looked at gerrit today, looking at it now
[16:44:15] <ebernhardson>	 should be just https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/979147
[16:45:56] <dcausse>	 so it's the first time we'll write to the update topic in kafka main I think
[16:46:06] <ebernhardson>	 yes i believe so. Do we need to repartition it first?
[16:46:19] <dcausse>	 might already be partitioned, lookihng
[16:47:25] <ebernhardson>	 looks like we have update.rc0 and fetch_error.rc0 topics in kafka-main with 5 partitions each
[16:47:43] <dcausse>	 ok :)
[16:48:35] <ebernhardson>	 i guess i should point the fetch-error-topic at .rc0 then...fixing
[16:49:45] <dcausse>	 ? should it be referencing the stream name?
[16:50:02] <ebernhardson>	 oh, indeed it does a lookup to decide it's rc0
[16:50:38] <ebernhardson>	 we have rc0 specific in the update-topic for other things though. Perhaps that shouldn't be?
[16:51:29] <dcausse>	 yes... hm.. perhaps it's because the job cannot decide which topic to use the codfw or the eqiad one and we have to help it
[16:51:58] <dcausse>	 hm I remember we implemented some kind of topic filtering for this
[16:54:03] <dcausse>	 yes we have topic-prefix-filter: 'eqiad' but might not be used for all the streams?
[16:55:05] <dcausse>	 hm but that's only for the producer...
[16:55:20] <ebernhardson>	 oh, i need to deploy producer in both dc's
[16:55:42] <dcausse>	 ebernhardson: I think you're right values-consumer-cloudelastic-eqiad.yaml should be updated with fetch-error-topic: eqiad.cirrussearch.update_pipeline.fetch_error.rc0
[16:58:48] <ebernhardson>	 ok, going to merge and start the producers first
[17:11:58] <ebernhardson>	 hmm. ran `kube_env cirrus-streaming-updater eqiad` and then `helmfile -e eqiad --selector name=producer -i apply`.    `kubectl get pods` says nothing is running, but helmfile diff doesn't say there is anything remaining to deploy
[17:13:11] <dcausse>	 the operator has to pick it up...
[17:13:15] <dcausse>	 might take time?
[17:13:44] <ebernhardson>	 it's been a couple minutes :S  wonder if we have to whitelist the namespace somehow
[17:14:21] <dcausse>	 oh indeed hm..
[17:15:03] <dcausse>	 in e.g. helmfile.d/admin_ng/values/eqiad/flink-operator-values.yaml
[17:15:24] <ebernhardson>	 oh! somehow i didn't notice there was a separate values directory and was wondering where it came from :)
[17:17:14] <ebernhardson>	 inflatador: if you have some time: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/992983
[17:19:36] <inflatador>	 ebernhardson :eyes
[17:20:33] <inflatador>	 ebernhardson +1'd, I can help deploy to admin-ng sometime this afternoon if you like
[17:20:59] <ebernhardson>	 inflatador: ok
[17:21:23] <inflatador>	 I'm heading to lunch but will reach out when I get back
[17:58:34] <ebernhardson>	 err, interesting...the "accepted" way to throw an exception from a session provider is to register a hook to either ApiBeforeMain or BeforeInitialize and throw from the closure you register
[18:02:19] * ebernhardson is tempted to try and move the exception throwing code from OAuth into core, so at least can re-use instead of copying, but i suspect then have to deal with additional concerns that aren't (currently) relevant
[18:13:07] <inflatador>	 back
[18:19:17] <inflatador>	 https://gerrit.wikimedia.org/r/c/operations/dns/+/993014 small CR for the cloudelastic migration if anyone has time to look
[18:48:15] <ebernhardson>	 inflatador: would appreciate deploying the admin-ng patch when you have a chance
[18:49:07] <inflatador>	 ebernhardson ACK, will ping you in 10-15 if that works
[18:50:29] <ebernhardson>	 sure, thanks
[19:19:31] <inflatador>	 ebernhardson I'm in https://meet.google.com/eki-rafx-cxi if you wanna join. Starting the process now
[20:46:15] * ebernhardson should have started on the producer yesterday, it's giving a fun `java.lang.OutOfMemoryError: Direct buffer memory` from the kafka producer.
[20:46:55] <ebernhardson>	 curiously, we configure `kafka-source-config.security.protocol: SSL` (and no kafka-sink-config, so it's copied) but the stacktrace includes org.apache.kafka.common.network.PlaintextTransportLayer.read
[20:49:37] <ebernhardson>	 huh, actually something doesn't work there. In the logs for taskmanager-1-1 it dumps the ProducerConfig before connecting, and the producer has `security.protocol = PLAINTEXT`. hmm
[21:00:14] <ebernhardson>	 setting that explicitly made things work, double checking the app config we only copy the bootstrap servers, not all kafka properties
[21:05:29] <inflatador>	 ERITF
[21:05:50] <inflatador>	 or should I say, "weird"
[21:05:53] <ebernhardson>	 :)
[21:06:24] <ebernhardson>	 i guess i had a wrong undertanding of the way we handle kafka config. I was thinking all kafka properties are copied from the source to sink if we don't define the sink, but it only copies the bootstrap list
[21:07:12] <ebernhardson>	 also noticed i didn't manage to properly limit the wikiids (only applied to eqiad, which is idle), so good thing only turned the producer on :) fixing...
[21:22:02] * ebernhardson now twiddles thumbs for 5 minutes waiting for the aggregation window
[21:34:10] <ebernhardson>	 sigh...5 minutes passed an no events coming out :S 
[21:34:59] <ebernhardson>	 and the answer is something i should have known... the configuration parser uses ; to separate lists and not ,
[21:36:17] <ryankemper>	 inflatador: patch to remove old cloudelastic masters => https://gerrit.wikimedia.org/r/c/operations/puppet/+/993038
[21:38:05] <inflatador>	 ryankemper ACK, +1'd
[21:45:06] <dr0ptp4kt>	 running queries serially without a pause against wdqs1022 and wdqs1024 from stat1006. doing a simple 5.5 minute test per node right now. i'll see what rps looks like, then want to ratchet it up a bit.
[21:52:50] <ryankemper>	 inflatador: alright gearing up to start working on the old masters
[21:53:00] <ryankemper>	 inflatador: can update you on irc or we can hop on a meet
[21:53:35] <inflatador>	 ryankemper meet is good, I'm up at https://meet.google.com/fde-tbpf-wqh
[22:23:31] <ebernhardson>	 alrighty, consumer-cloudelastic is up and running, writing to testwiki and mediawikiwiki.  Expecting it to basically do nothing, since the old updater is also running and has lower end-to-end latency
[22:57:30] <ryankemper>	 (off-topic) I'm going to see Lotus (jam band) in LA for the first time tonight. Here's a song of theirs I like (also good for work since it's minimal vocals): https://www.youtube.com/watch?v=iMS8k7pkmlM
[23:20:02] <inflatador>	 it's smooooth
[23:20:40] <dr0ptp4kt>	 at first i was expecting Flying Lotus, but no, this is Lotus. cool! have fun!
[23:53:30] <ryankemper>	 Flying lotus has great stuff too for sure!