[07:01:03] <dcausse>	 pfischer: (if not done already) should we file a ticket to ask for topic creation?
[07:10:12] <dcausse>	 iiuc we'll need 3 topics per DC: $DC.cirrussearch.update_pipeline.fetch_error.rc0, $DC.cirrussearch.update_pipeline.update.rc0 and $DC.mediawiki.cirrussearch.page_rerender.v1
[07:24:38] <pfischer>	 dcausse: good point, I’ll do that. That should be tagged with service ops? From the versioning scheme, I guess we keep page_rerender out of the development space (for schemas).
[07:25:13] <dcausse>	 yes, added the v1 suffix
[07:26:10] <dcausse>	 I thought that the schema being simple enough minor and compatible changes will be possible
[07:26:32] <pfischer>	 Sure, I’ll request deployment for this afternoon then: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20230926T1300
[07:27:00] <dcausse>	 the page_rerender one requires a patch to cirrus search sadly
[07:27:10] <dcausse>	 so will have to wait for the train
[07:27:35] <dcausse>	 pfischer: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/961000
[07:30:51] <dcausse>	 but yours does not have to wait for the train I think
[07:33:29] <dcausse>	 also we might have to request a restart of evengate-main after new streams are deployed to mw-config (c.f. https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration#Stream_Configuration_deployment)
[08:01:03] <pfischer>	 Yes, although I do not understand the limitation to streams “that will be produced through an destination_event_service” in the end. Is this relevant? To my understanding both SUP streams (update and fetch_error) are not produced from EventGate, therefore, we do not need to specify destination_event_service.
[08:02:33] <pfischer>	 Or do we replicate them to HDFS and need canary events to make sure we don’t have empty buckets?
[08:02:47] <dcausse>	 pfischer: indeed, canary events would have been the exception but we don't activate them apparently
[08:04:09] <dcausse>	 hm.. thinking about canary events I think that only the fetch_error stream might be consumed from hdfs
[08:05:22] <dcausse>	 we might need the canary events for cases when there are no fetch errors within an hour 
[08:05:44] <dcausse>	 so perhaps we should enable canary events for all our streams?
[08:07:49] <dcausse>	 and set destination_event_service to eventgate-main (for canary events) eventhough we're producing directly to kafka?
[08:12:19] <pfischer>	 Sure, I’ll update my patch
[09:22:47] <dcausse>	 pfischer: I'd like to make a release of the rdf repo possibly with https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/960554 (thanks for the +1!), are there things you're not sure we could discuss to +2 this patch? (totally fine to wait for Erik's review if you prefer tho)
[09:32:43] <dcausse>	 errand + lunch
[09:59:09] <gehel>	 Lunch 
[10:37:38] <pfischer>	 dcausse: 👀
[10:38:29] <pfischer>	 dcausse: +2
[10:41:54] <pfischer>	 errands + lunch
[12:09:06] <dcausse>	 thanks for the +2!
[12:41:25] <pfischer>	 gehel: There are some low hanging SUP-related ticket fruits on our work board: https://phabricator.wikimedia.org/project/board/1849/. Since we skipped triage: How can we pull in tickets mid-week? Should I just estimate them myself?
[12:41:48] <gehel>	 yep, just estimate them yourself and pull them in!
[13:09:10] <dcausse>	 pfischer: deploy window started in #wikimedia-operations
[13:12:45] <dcausse>	 pfischer: amended your patch based on latest comments
[13:13:32] <inflatador>	 <o/
[13:13:57] <dcausse>	 o/
[14:03:44] <gehel>	 dcausse, inflatador : SUP meeting in https://meet.google.com/pup-xwxi-oqw
[14:39:40] <dcausse>	 inflatador: any objections if I try to deploy https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/961101 and https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/959218 to wikikube-staging (context is T326914)
[14:39:41] <stashbot>	 T326914: Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink - https://phabricator.wikimedia.org/T326914
[14:40:20] <inflatador>	 dcausse no, feel free to deploy. I'm free for the next 20m if you need anything
[14:40:29] <dcausse>	 thanks!
[15:01:52] <ebernhardson>	 \o
[15:06:03] <inflatador>	 <o/
[15:06:25] <dcausse>	 o/
[15:19:31] <pfischer>	 o/
[15:21:14] <dcausse>	 forgot something regarding moving away from FlinkKafkaConsumer with the WDQS flink job, if anyone has a couple minutes: https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/961137
[15:22:28] <ebernhardson>	 dcausse: i don't know what it does, but it looks plausible :) 
[15:22:47] <dcausse>	 ebernhardson: this is from this procedure: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kafka/#upgrading-to-the-latest-connector-version
[15:23:00] <dcausse>	 thanks! :)
[15:37:01] <pfischer>	 dcausse: just found a nit
[15:37:05] <dcausse>	 sigh... seems like I forgot something else... KafkaSource defaults to earliest not group offsets :(
[15:37:36] <dcausse>	 pfischer: thanks!
[15:39:16] <pfischer>	 Off, back in 60’
[15:48:46] <inflatador>	 workout, back in ~40
[15:55:56] <dcausse>	 and another one: https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/961144, sorry about that I should have paid more attention here and not assume that defaults would be the same...
[15:57:02] <ebernhardson>	 yup, seems sane. Curious they changed that default
[15:57:14] <ebernhardson>	 i suppose, i think pykafka does the same with defaulting to earliest
[15:57:25] <dcausse>	 perhaps to avoid failure when starting a fresh app?
[15:57:37] <ebernhardson>	 yea perhaps
[15:58:03] <ebernhardson>	 i guess pykafka only defaults to earliest if there are no commited offsets though, it's odd it would ignore them
[15:58:28] <dcausse>	 yes true
[16:30:49] <inflatador>	 back
[16:44:23] <inflatador>	 if you still need reviews for the WDQS deploy stuff LMK
[16:45:20] <dcausse>	 inflatador: thanks, still waiting for CI so might not get to it today, (still will have to make a new release after that)
[16:45:55] <inflatador>	 dcausse np, I know it's late for you
[16:51:27] <dcausse>	 yes, so the /srv/deployment/wdqs/wdqs state is not great at that point as it does not reflect what's running, if there's a need to re-deploy this flink job (I highly doubt we'll need to but...) the few patches merged to the deploy repo today will have to be reverted.
[17:01:07] <inflatador>	 updated partman recipe ready for review if anyone has time: https://gerrit.wikimedia.org/r/c/operations/puppet/+/960114/
[17:01:31] <inflatador>	 Not sure if it'll work, but it shouldn't mess with anyone else's servers at least
[17:02:34] <ebernhardson>	 inflatador: seems plausible
[17:03:21] <inflatador>	 ebernhardson thanks! Will give it a shot shortly
[17:15:10] <inflatador>	 and one more patch for adding cloudelastic to site.pp: https://gerrit.wikimedia.org/r/c/operations/puppet/+/961167
[17:16:06] <ebernhardson>	 inflatador: is your cloudelastic patch intending to change the cloudcontrol ones?
[17:17:35] <inflatador>	 ebernhardson no...not sure WTF happened there...will fix
[17:17:50] <dcausse>	 dinner
[17:23:22] <inflatador>	 hmm, getting `empty range in char class: /^cloudelastic10[07-12]\.wikimedia\./`...bad regex or more likely I missed a lifecycle step. Checking...
[17:23:56] <ebernhardson>	 hmm
[17:24:16] <ebernhardson>	 oh, because 07-12 is not 07,08,09,10,11,12
[17:24:27] <ebernhardson>	 its 0, 2 and 7-1 (which is invalid)
[17:24:41] <ebernhardson>	 inflatador: ^
[17:25:04] <inflatador>	 ebernhardson got it, looking at other host regexes
[17:25:51] <ebernhardson>	 i think we've often kept it simple, make two rules one for cloudelastic100[7-9] and another for cloudelastic101[0-2]
[17:26:02] <ebernhardson>	 otherwise regex alternation can do it, but it looks funky
[17:36:12] <inflatador>	 I was pinging in dcops as well, looks like rob-h went ahead and merged my fixed patch
[17:44:52] <inflatador>	 oops, added too many hosts...there's only 07-10. New patch up to fix: https://gerrit.wikimedia.org/r/c/operations/puppet/+/961173
[17:45:18] <ebernhardson>	 lol :)
[17:45:48] <inflatador>	 lunch, back in time for SRE pairing
[18:21:01] <inflatador>	 back
[20:28:51] <inflatador>	 break, back in ~15
[20:45:50] <ebernhardson>	 sigh..spark is just odd sometimes.  It was reading the input as 3 partitions and OOM'ing, so tell it to limit partitions to 16MB when reading and it increased to 12 partitions, great! Except 8 of those partitions are empty :P
[20:46:45] <ebernhardson>	 and it still has the same 2 large partitions as before...sigh. 
[20:53:54] <inflatador>	 back
[20:54:10] <inflatador>	 sounds strangely similar to partman ;)