[00:17:43] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: [Event Platform] eventutilites-python: improve consistency guarantees of async process functions - https://phabricator.wikimedia.org/T347282#11780618 (10Ottomata) [03:04:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Backfill newly productionized edit types dataset - https://phabricator.wikimedia.org/T421919#11780701 (10fkaelin) There is a notebooks folder in research-datasets, you could put a notebook there too. Running this at scale in a notebook is unpleasant. Y... [06:42:25] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Traffic: Surge in webrequest sequence-id validation check - https://phabricator.wikimedia.org/T422030#11780870 (10JAllemandou) [06:42:36] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Traffic: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11780871 (10JAllemandou) [07:04:42] 06Data-Engineering, 06MW-Interfaces-Team, 06Traffic, 07OKR-Work: Log Api-User-Agent header in Turnilo - https://phabricator.wikimedia.org/T373871#11780894 (10daniel) Having this would also allow us to decide whether it makes sense to start using Api-User-Agent as an alternative to the normal User-Agent hea... [07:41:15] 06Data-Engineering, 06Data-Engineering-Radar, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11780933 (10gkyziridis) === Update === The [[ https://gerrit.wikimedia.org/r/plugins/g... [09:39:58] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Data-Engineering-Wikistats, 10Wikidata: Wikidata unique devices statistics are obviously wrong - https://phabricator.wikimedia.org/T420210#11781262 (10TTWIDEE) This also appears to be the case for Wikifunctions: https://stats.wikimedia.org/#/wikifun... [10:52:52] 06Data-Engineering, 10Data-Engineering-Wikistats: Add total file size to metric to Wikistats - https://phabricator.wikimedia.org/T421598#11781753 (10GGoncalves-WMF) A couple of notes about this use case, as far as I can tell: - [[ https://en.wikipedia.org/wiki/Template:Wikipedia_article_graph | Template:Wikip... [11:40:08] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11781934 (10Gehel) p:05Triage→03High [11:40:38] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11781938 (10Gehel) @Ahoelzl : could you validate this access request? [11:43:28] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783#11781940 (10Gehel) Since the user is already in the appropriate groups, I assume that access was already reviewed and this is just a technical step. No further... [13:17:13] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team: CentralAuth's localuser table contains many nulls and duplicate mappings - https://phabricator.wikimedia.org/T411116#11782327 (10APizzata-WMF) The table has been sqooped and I have ran all... [14:19:31] !log Test Kitchen edge-unique experiments (poll 59497) - adds: none; removes: none; fields: logged-out-retention-round5 - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:19:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:30:16] !log Test Kitchen edge-unique experiments (poll 59529) - adds: none; removes: attribution-research-short-baseline-run; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:30:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:10:00] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Backfill datasets affected by Nov 2025 automated traffic incident - https://phabricator.wikimedia.org/T421735#11782918 (10mforns) [15:22:58] 06Data-Engineering, 06Data-Engineering-Radar, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11782975 (10gkyziridis) === Update === We reverted the the changes on production becau... [15:24:02] 06Data-Engineering, 06Data-Engineering-Radar, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11782977 (10gkyziridis) a:03gkyziridis [15:37:13] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Fix PyFlink log levels - https://phabricator.wikimedia.org/T419997#11783051 (10JMonton-WMF) Another main Error that appears as INFO, just right after a real ERROR: `json { "_index": "ecs-k8s-1-1.11.0-7-2026.14", "_id": "G3jBTp0BDa... [16:31:59] 06Data-Engineering, 06Product-Analytics (Kanban): Follow-up analysis to understand usage of dumps to inform v2 rollout - https://phabricator.wikimedia.org/T402963#11783326 (10HCoplin-WMF) 05Open→03Resolved Apologies for the delay! Marking as resolved. Analysis was comprehensive and very useful for info... [16:47:02] 06Data-Engineering, 10SRE-Access-Requests: Update production access key for ptiwary - https://phabricator.wikimedia.org/T422189 (10prabhat) 03NEW [16:47:47] 06Data-Engineering, 10SRE-Access-Requests: Update production access key for ptiwary - https://phabricator.wikimedia.org/T422189#11783451 (10prabhat) [16:52:26] 06Data-Engineering, 10SRE-Access-Requests: Update production access key for ptiwary - https://phabricator.wikimedia.org/T422189#11783519 (10ssingh) request and key confirmed out of band. [17:05:29] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 05MW-1.46-notes (1.46.0-wmf.22; 2026-03-31), and 2 others: Deprecate and remove mw.eventLog.submitClick() - https://phabricator.wikimedia.org/T415210#11783619 (10KReid-WMF) 05Open→03Resolved [17:05:55] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 07Essential-Work, and 2 others: Deluge of inactionable console warnings - https://phabricator.wikimedia.org/T419481#11783629 (10KReid-WMF) 05Open→03Resolved [17:07:10] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 10Test Kitchen (Test Kitchen (Experiment Platform Sprint 22)): Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11783657 (10KReid-WMF) [17:07:20] 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10MediaWiki-extensions-WikimediaEvents, and 5 others: Could not hoist data into experiment.subject_id for event - https://phabricator.wikimedia.org/T421152#11783649 (10KReid-WMF) [17:26:28] 06Data-Engineering, 10SRE-Access-Requests: Update production access key for ptiwary - https://phabricator.wikimedia.org/T422189#11783746 (10prabhat) [17:35:10] 06Data-Engineering, 06Data-Engineering-Radar, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11783778 (10Ottomata) > changeprop errors Weird! These indeed look like some logging... [17:37:07] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Fix PyFlink log levels - https://phabricator.wikimedia.org/T419997#11783788 (10Ottomata) > Another main Error that appears as INFO, just right after a real ERROR: I think this one is correctly INFO. The message is about the applicati... [17:44:27] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Update production access key for ptiwary - https://phabricator.wikimedia.org/T422189#11783815 (10HShaikh) As prabhat's manager I approve this request. [17:53:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:53:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:58:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:58:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:23:36] RESOLVED: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:23:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:25:06] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:25:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:29:03] FIRING: [6x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:29:09] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:19:43] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Backfilling configuration - https://phabricator.wikimedia.org/T421216#11784160 (10Ottomata) @JMonton-WMF and @AKhatun_WMF while we have backfill tuning issues, I want the production job to run, n... [19:21:36] 06Data-Engineering, 10Data-Engineering-Wikistats, 10Pageviews-Anomaly: Sudden traffic increase on 1 November 2025 - https://phabricator.wikimedia.org/T412655#11784162 (10Oesjaar) Thank you for the feedback . Appreciated [19:33:06] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Update production access key for ptiwary - https://phabricator.wikimedia.org/T422189#11784179 (10ssingh) 05Open→03Resolved a:03ssingh Should now be rolled out everywhere, let us know if you have any issues with access. [19:38:39] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: HTML Enrichment - Backfilling configuration - https://phabricator.wikimedia.org/T421216#11784194 (10Ottomata) [19:38:50] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: HTML Enrichment - Backfilling configuration - https://phabricator.wikimedia.org/T421216#11784195 (10Ottomata) [19:39:11] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11784199 (10Ottomata) [20:21:32] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11784282 (10Ottomata) Okay, in staging (-next) I just applied ` # start from timestamp for backfill test: # 1774828800000 == Monday, Marc... [20:26:03] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11784294 (10Ottomata) @JMonton-WMF good luck to you! Things we need to try: - Increase mediawiki.page_change.v1 kafka topic partitions... [22:29:03] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:29:09] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag