[08:30:36] <bartosz>	 o/ giving a little heads up that I deployed changeprop on staging/codfw/eqiad to enable more wikis for one of our ML services (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1328405/). Everything is looking healthy in all DCs, excuse me for not pinging beforehand! 
[08:57:00] <_joe_>	 changeprop, or changeprop-jobqueue?
[08:57:28] <_joe_>	 ah, changeprop. I wish we had a better solution that expanding its use by now :(
[09:06:12] <bartosz>	 lately there are discussions and work around Linked Artifacts Cache, which could let us move away from changeprop, so there is hope! :) https://phabricator.wikimedia.org/T432733 https://phabricator.wikimedia.org/T433699
[12:52:48] <brouberol>	 Hi serviceops! I'm planning to increase the number of partitions for the `{eqiad,codfw}.mediawiki.job.htmlCacheUpdate` topics from 2 to 6. These seem to be related to changeprop. Do you know if it should simply reconfigure its consumer group with the new # of partitions, or if we should restart it? If it's the latter, could we sync up to make sure
[12:52:48] <brouberol>	 there's no hiccup? Thanks!
[14:50:46] <swfrench-wmf>	 brouberol: changeprop uses a pretty simple consumer group / subscription mechanism for the job topics of which it's aware (e.g., doesn't do wild stuff with assign anything). thus, my expectation is that the new partitions should automatically get picked up by a rebalance.
[14:52:39] <swfrench-wmf>	 once you make the change, and we see messages produced by eventgate flowing into the new topic partitions, we should probably confirm the expected rebalance happens and we see the consumer group offset advancing on them.
[14:55:22] <brouberol>	 understood! Thanks
[14:56:49] <swfrench-wmf>	 brouberol: ... but to dig a little deeper into your question: this particular job is a little weird - in that what changeprop does is actually steps. the {eqiad,codfw}.mediawiki.job.htmlCacheUpdate consumer doesn't directly process the job, and instead just delegates them to _other_ topics with manual partitioning
[14:57:06] <swfrench-wmf>	 those are the {eqiad,codfw}.cpjobqueue.partitioned.mediawiki.job.htmlCacheUpdate partitions
[14:57:10] <swfrench-wmf>	 s/partition/stopics/
[14:57:24] * swfrench-wmf cannot type yet today, aparently
[14:58:03] <brouberol>	 for context I'm seeing a surge of activity on that topic that causes the partitions to grow larger than we'd like https://grafana.wikimedia.org/d/000000234/kafka-by-topic?orgId=1&from=now-24h&to=now&timezone=utc&var-datasource=000000006&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=%24__all&var-topic=eqiad.mediawiki.job.htmlCacheUpdate
[14:59:06] <swfrench-wmf>	 uhhh ... hmmm ... why is this in jumbo? changeprop doesn't consume from that cluster :)
[15:00:12] <brouberol>	 it might just be mirrored 
[15:00:25] <brouberol>	 and in that case, the "real" producer is mirrormaker
[15:00:33] <swfrench-wmf>	 agreed, yeah
[15:00:47] <swfrench-wmf>	 so, if you're just touching jumbo, then you do not need to worry about changeprop!
[15:00:55] <brouberol>	 great stuff, thanks
[15:01:15] <swfrench-wmf>	 thanks for checking!
[15:01:36] <brouberol>	 !log brouberol@kafka-jumbo1010:~$ sudo -E kafka topics --alter --topic eqiad.mediawiki.job.htmlCacheUpdate --partitions=6
[18:05:21] <bd808>	 swfrench-wmf: I don't think this is a problem , but I thought I would share with you that the shellbox-video updates are removing a `FPM__log_level: debug` envvar as they roll out.
[18:05:55] <swfrench-wmf>	 ah! I guess that never got rolled out ... not a problem, and thank you for flagging!