[00:25:22] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform, 13Patch-For-Review: Event stream with latest revision HTML & parent revision HTML diff - https://phabricator.wikimedia.org/T360794#11784786 (10Ottomata) @JMonton-WMF something we should keep an eye on: kafka topic size... [00:32:35] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add support for variables to DbtSkeinOperator - https://phabricator.wikimedia.org/T421789#11784793 (10amastilovic) @Mayakp.wiki yes this is for the backfill functionality, among other stuff! [01:08:17] 06Data-Engineering, 06Data-Engineering-Icebox, 06DBA, 13Patch-For-Review: Move Mostcategories computation to Hadoop - https://phabricator.wikimedia.org/T413362#11784863 (10Zabe) Taking a look at https://analytics.wikimedia.org/published/datasets/querypage/MostCategories/commonswiki.json and comparing it to... [01:13:55] 06Data-Engineering, 06Data-Engineering-Icebox, 06DBA, 13Patch-For-Review: Move Mostcategories computation to Hadoop - https://phabricator.wikimedia.org/T413362#11784867 (10Zabe) Ok, the difference is that the MediaWiki implementation filters for pages in `$wgContentNamespaces` and this includes files on co... [02:29:03] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:29:09] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:00:06] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:00:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:18:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:18:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [05:23:51] RESOLVED: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [05:23:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:42:05] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform, 13Patch-For-Review: Event stream with latest revision HTML & parent revision HTML diff - https://phabricator.wikimedia.org/T360794#11785032 (10brouberol) @Ottomata Assuming you mean 290GB and not 290TB, we should be all... [11:16:58] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11785408 (10brouberol) a:03brouberol [11:17:01] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11785410 (10brouberol) 05Open→03In progress [11:17:40] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11785411 (10brouberol) ` brouberol@krb1002:~$ sudo manage_principals.py create matmarex --email=bdziewonski@wikimedia.org Principal already created (or an erro... [11:18:02] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11785412 (10brouberol) It appears as though you already have a kerberos principal created @matmarex [11:18:14] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11785415 (10brouberol) 05In progress→03Resolved [13:28:33] !log Test Kitchen edge-unique experiments (poll 63615) - adds: logged-out-retention-round6; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [13:28:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:44:53] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Implement list of JA3N-JA4H pairs to be tagged as automated into the bot detection pipeline - https://phabricator.wikimedia.org/T420412#11785675 (10mforns) I've finished the tests. Hamid and I have checked that both actor counts and pageview counts for di... [15:24:14] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11785782 (10matmarex) 05Resolved→03Open That's a surprise. In that case, may I ask for its password to be reset? (per 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11785827 (10brouberol) Sure thing! ` brouberol@krb1002:~$ sudo manage_principals.py reset-password matmarex --email=bdziewonski@wikimedia.org Password reset su... [16:42:18] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions, 13Patch-For-Review: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11786005 (10Ahoelzl) @dcausse can you also confirm that the latest April run / data was... [17:51:20] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Commons, 06Data-Persistence, 07Epic, and 2 others: FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster - https://phabricator.wikimedia.org/T398709#11786217 (10Ahoelzl) [17:52:01] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Traffic referrer analysis - https://phabricator.wikimedia.org/T421516#11786223 (10Ahoelzl) [18:31:38] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Reader Experience Team, 10Test Kitchen, 05MW-1.46-notes (1.46.0-wmf.22; 2026-03-31): Logged in reader retention logging - https://phabricator.wikimedia.org/T420621#11786329 (10tchin) Data is now available in the data lake under `wmf_readership.act... [18:37:51] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Observability-Metrics: [Data Quality] Sending Apache Spark metrics to PushGateway - https://phabricator.wikimedia.org/T297231#11786340 (10Ottomata) Cool! Seems easy enough, except last commit is 7 years ago? :D [18:40:16] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11786345 (10Ottomata) Looks like prod died, and is now backfilling! Ah, but if there is no checkpointed offsets, fl... [18:48:26] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11786351 (10matmarex) 05Open→03Resolved Thanks! [19:23:16] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Implement list of JA3N-JA4H pairs to be tagged as automated into the bot detection pipeline - https://phabricator.wikimedia.org/T420412#11786393 (10mforns) I prepared a deployment plan for the automated traffic detection changes: 1.... [20:09:31] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11786474 (10Ottomata) Well! Staging is failing with message too large in kafka sink again: > Caused by: org.apache.kafka.common.errors.R... [20:13:27] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11786485 (10Ottomata) For now, in staging, I'm going to reduce `enrich.max_content_size` to 15MB, giving us a 5MB margin. ` helmfile app... [21:33:29] (03PS1) 10Zabe: querypage: mostcategories: Include NS_FILE if running on commons [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1267966 (https://phabricator.wikimedia.org/T413362)