[01:18:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11787170 (10Ottomata) The production job has started getting stuck. https://grafana.wikimedia.org/goto/bfi0ytdvi6y2od?orgId=1 It hasn't fai... [12:39:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11787480 (10Ottomata) Huh! And production caught back up! It did not crash or restart. I do see that about 10:00 UTC today TM 1-1's Youn... [12:56:51] PROBLEM - statsv Varnishkafka log producer on cp4042 is CRITICAL: PROCS CRITICAL: 3 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [12:57:51] RECOVERY - statsv Varnishkafka log producer on cp4042 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [15:22:14] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11787597 (10Ottomata) > Increase mediawiki.page_change.v1 kafka topic partitions and increase kafka source parallelism @JMonton-WMF this is s... [17:59:50] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11787683 (10Ottomata) FYI, we see log messages like ` Name collision: Group already contains a Metric with the name 'pendingCommittables'. Me... [18:18:28] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11787705 (10Ottomata) Ah, I figured out why the Flink UI wasn't showing metrics. `metrics.internal.query-service.port`... [18:50:00] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11787728 (10Ottomata) > Can we do better than 400ms in the normal case? So, according to MW REST API latencies, the p5... [20:43:51] 06Data-Engineering, 10Dumps-Generation, 10Wikidata: Json wikidata dumps incomplete - https://phabricator.wikimedia.org/T422303 (10Melderick) 03NEW [21:39:23] 06Data-Engineering, 10Dumps-Generation, 10Wikidata: Json wikidata dumps incomplete - https://phabricator.wikimedia.org/T422303#11787815 (10So9q) and perhaps set up an alarm to make sure you get a notice if they are too small in the future?