[11:37:49] <dcausse>	 lunch
[14:15:58] <inflatador>	 <o/
[14:29:53] <dcausse>	 o/
[17:27:58] <inflatador>	 re: flink alerts for the new SUP, looks like there is a {{ $labels.release }} we can use for our alerts
[18:03:10] <inflatador>	 lunch, back in ~1h
[18:49:00] <inflatador>	 back
[19:35:25] <inflatador>	 Looking at helm releases for the SUP. Looks like we have consumer, consumer backfill, and producer. Is that all of 'em? And do we plan on adding more? just looking at the alert stuff
[19:51:15] <ebernhardson>	 inflatador:  for now there should be 1 producer per datacenter, 1 consumer per cluster in the datacenter, and then a backfill that may or may not exist per cluster in the datacenter
[19:51:20] <ebernhardson>	 i don't think we have plans for any more
[19:51:38] <ebernhardson>	 where cluster == cirrus cluster (group of 3 elastic clusters)
[19:53:19] <inflatador>	 Got it
[20:19:23] * ebernhardson realizes he has no way to add new wikis when they start existing
[20:19:34] <ebernhardson>	 can write something, but always the little details :P
[22:11:06] <inflatador>	 monitoring sporadic jobs like backfill will be a little different. I guess we'll need a similar approach as we talked about with airflow?
[23:35:33] <ebernhardson>	 i don't know how effectively you can monitor sporadic jobs, you can't really assert much about them
[23:35:55] <ebernhardson>	 you instead end up depending on tooling around running the sporadic things to report errors