[00:43:05] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Monthly reconcile continues to emit a really large amount of events after user_id changes - https://phabricator.wikimedia.org/T419055#11700415 (10xcollazo) This is the state before allowing `spark_emit_reconcile_events_to_kafka` to r... [00:44:35] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Monthly reconcile continues to emit a really large amount of events after user_id changes - https://phabricator.wikimedia.org/T419055#11700425 (10xcollazo) >>! In T419055#11699373, @xcollazo wrote: > Double checked that `mw_content_r... [00:58:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:58:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [01:13:47] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0, 13Patch-For-Review: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11700442 (10Eevans) >>! In T415202#11689497, @Snwachukwu wrote: > @AndrewTavis_WMDE @Ladsgroup , I’ve put together a design [[... [02:08:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:08:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:13:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:13:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:13:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:13:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [08:24:30] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Research-engineering, 06Research-Freezer, 10Event-Platform, 13Patch-For-Review: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#11701020 (10brouberol) ` brouberol@cephosd1001:~$ sudo radosgw-admin user create --uid=mw-page-... [08:41:36] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: On reconcile, consider what happens when a restore and a delete happen on the same revision - https://phabricator.wikimedia.org/T412461#11701121 (10APizzata-WMF) Thanks to the backup @xcollazo created in T419055#11699345 we can... [09:49:27] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Dumps-Generation: Data missing from en.wiktionary.org February 2026 "MediaWiki Content File Exports" compared to "XML Database dump" - https://phabricator.wikimedia.org/T417596#11701310 (10APizzata-WMF) In this [[ https://gitlab.wikimedia.org/repos/da... [10:10:28] 06Data-Engineering, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 06Growth-Team (FY2025-26 Q3 Sprint 5): New accounts created from editor anon warning redirect to welcome survey, not back to editor (2026) - https://phabricator.wikimedia.org/T419723#11701365 (10Urbanecm_WMF) I reproduced th... [10:11:43] (03CR) 10Aqu: [C:03+2] Move bot detection pipeline into new repo [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1237928 (https://phabricator.wikimedia.org/T415874) (owner: 10Aqu) [10:12:17] (03CR) 10Aqu: [V:03+2 C:03+2] Move bot detection pipeline into new repo [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1237928 (https://phabricator.wikimedia.org/T415874) (owner: 10Aqu) [10:13:34] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11701372 (10Ladsgroup) Very likely we will just announce the deprecation and won... [10:13:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:13:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:32:28] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Use wmf.mediawiki_history as baseline for slo completeness - https://phabricator.wikimedia.org/T416312#11701420 (10APizzata-WMF) Everything has been merged, will test run the monthly pipeline once the [[ https://airflow.wikimedi... [10:48:59] 06Data-Engineering, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 06Growth-Team (FY2025-26 Q3 Sprint 5): New accounts created from editor anon warning redirect to welcome survey, not back to editor (2026) - https://phabricator.wikimedia.org/T419723#11701461 (10Tgr) We shouldn't sample erro... [11:02:33] !log Test Kitchen edge-unique experiments (poll 243589) - adds: mobile-toc-abc2; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:02:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:11:52] 06Data-Engineering, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 06Growth-Team (FY2025-26 Q3 Sprint 5): New accounts created from editor anon warning redirect to welcome survey, not back to editor (2026) - https://phabricator.wikimedia.org/T419723#11701531 (10Tgr) >>! In T419723#11700046,... [11:13:06] 06Data-Engineering, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 06Growth-Team (FY2025-26 Q3 Sprint 5): New accounts created from editor anon warning redirect to welcome survey, not back to editor (2026) - https://phabricator.wikimedia.org/T419723#11701537 (10Urbanecm_WMF) I discussed thi... [11:20:07] 06Data-Engineering, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 06Growth-Team (FY2025-26 Q3 Sprint 5): New accounts created from editor anon warning redirect to welcome survey, not back to editor (2026) - https://phabricator.wikimedia.org/T419723#11701549 (10Urbanecm_WMF) In `AuthManager... [11:54:38] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11701643 (10GGoncalves-WMF) I'm wondering how we would compare this with the API Analytics Dashboard, and the underlying data product we want... [12:59:07] 06Data-Engineering, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 06Growth-Team (FY2025-26 Q3 Sprint 5): New accounts created from editor anon warning redirect to welcome survey, not back to editor (2026) - https://phabricator.wikimedia.org/T419723#11701879 (10Tgr) I believe this is caused... [12:59:09] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Do multiple code and data clean ups for content tables - https://phabricator.wikimedia.org/T418754#11701880 (10xcollazo) ` spark.sql(""" SELECT committed_at, snapshot_id, operation, summary['added-files-size']/10... [13:11:10] 06Data-Engineering, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 06Growth-Team (FY2025-26 Q3 Sprint 5): New accounts created from editor anon warning redirect to welcome survey, not back to editor (2026) - https://phabricator.wikimedia.org/T419723#11702001 (10Michael) >>! In T419723#11701... [14:05:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0, 13Patch-For-Review: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11702303 (10Snwachukwu) @Eevans thank you for taking a look at the design doc. We decided to reuse existing mediarequest Cassa... [14:13:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:13:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:07:21] 06Data-Engineering, 06Data-Engineering-Radar, 07Essential-Work, 10Event-Platform, and 2 others: X-Experiment-Enrollments EventGate handling reinforcement for MalformedHeaderError cases - https://phabricator.wikimedia.org/T409106#11702693 (10KReid-WMF) 05Open→03Resolved [15:07:30] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 07Technical-Debt, 06Test Kitchen (Experiment Platform Sprint 20): Decouple mw.xLab.getInstrument() and mw.eventLog.newInstrument() - https://phabricator.wikimedia.org/T408094#11702698 (10KReid-WMF) 05Open→03Resolved [15:10:58] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 06Test Kitchen (Experiment Platform Sprint 21): Deluge of inactionable console warnings - https://phabricator.wikimedia.org/T419481#11702743 (10KReid-WMF) [15:25:07] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 06Test Kitchen (Experiment Platform Sprint 21): Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11702850 (10KReid-WMF) [15:46:09] 06Data-Engineering, 10AQS2.0: Review and adapt wikistats / AQS video analytics to the introduction of MPEG-DASH - https://phabricator.wikimedia.org/T419879 (10GGoncalves-WMF) 03NEW [15:51:47] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for ben.buchenau - https://phabricator.wikimedia.org/T390734#11703061 (10Ben.buchenau) Thanks for all the clarification! In that case, being added to `analytics-wmde-users` sounds like the best way to go... [15:52:09] 06Data-Engineering, 10AQS2.0: Consider updating our heuristics for media type classification in AQS / wikistats - https://phabricator.wikimedia.org/T419882 (10GGoncalves-WMF) 03NEW [15:53:35] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for ben.buchenau - https://phabricator.wikimedia.org/T390734#11703092 (10Dzahn) 05Open→03Resolved [15:58:24] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0, 13Patch-For-Review: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11703123 (10GGoncalves-WMF) Thanks for the context, @TheDJ ! I've filed T419879 to address measurement issues with the introdu... [16:18:10] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Logs and Monitoring for the HTML pipeline - https://phabricator.wikimedia.org/T418996#11703201 (10JMonton-WMF) Another interesting example, it looks like logs defined in Python are picked by Flink and reported as INFO, not matter if th... [16:45:26] 06Data-Engineering, 10Data-Services, 06cloud-services-team (FY2025/2026-Q3-Q4), 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11703287 (10fnegri) 05Open→03In progress p:05Triage→03Medium a:... [16:53:17] 06Data-Engineering, 10Data-Services, 06cloud-services-team (FY2025/2026-Q3-Q4), 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11703319 (10fnegri) I ran this on all clouddbs: ` sudo maintain-views... [16:55:10] 06Data-Engineering, 10Data-Services, 06cloud-services-team (FY2025/2026-Q3-Q4), 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11703330 (10fnegri) a:05fnegri→03None [17:17:35] !log Test Kitchen edge-unique experiments (poll 244706) - adds: synth-test-new-external-path; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [17:17:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:10:54] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add Fixture Generation for KubernetesPodOperator Tasks - https://phabricator.wikimedia.org/T419889 (10xcollazo) 03NEW [18:13:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:13:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:13:51] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:13:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:07:16] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857#11704747 (10MusikAnimal) In recent tests, it seems to start giving 429s after exactly 1000 requests.... [23:16:54] !log Test Kitchen edge-unique experiments (poll 245777) - adds: share-highlight-baseline; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [23:16:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log