[01:53:51] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11974851 (10AKhatun_WMF) Details/Caveats of this dataset: - We are using `event_user_text_historical`, meaning user name changes are not... [07:41:20] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create API and User-Agent compliance related tables under wmf_traffic - https://phabricator.wikimedia.org/T427840#11975217 (10JAllemandou) >>! In T427840#11973679, @KCVelaga_WMF wrote: > Also, for future use cases, if it isn't an issue, can we consider enabl... [07:49:53] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create API and User-Agent compliance related tables under wmf_traffic - https://phabricator.wikimedia.org/T427840#11975266 (10JAllemandou) @KCVelaga_WMF : I don't understand why you use explicit (year, month, day, hour) partitioning for those tables when we... [07:57:22] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create API and User-Agent compliance related tables under wmf_traffic - https://phabricator.wikimedia.org/T427840#11975279 (10JAllemandou) I have created the folders for the tables, but not schemas, I'm waiting for the discussion over Iceberg partitioning. [08:11:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11975297 (10JAllemandou) > Cross-wiki users are marked as anonymous. This may be misleading as the users are not really anonymous. These... [08:20:52] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11975313 (10APizzata-WMF) Yesterday's sqoop process terminated ingesting all the necessary... [08:22:00] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create API and User-Agent compliance related tables under wmf_traffic - https://phabricator.wikimedia.org/T427840#11975333 (10KCVelaga_WMF) >>! In T427840#11975266, @JAllemandou wrote: > @KCVelaga_WMF : I don't understand why you use explicit (year, month, d... [12:39:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [12:39:42] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [12:44:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [12:44:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [12:49:36] RESOLVED: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [12:49:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:10:00] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11976366 (10Ottomata) > This means if someone was deemed bot later on (or vice versa), we won't know that until that user makes a new cha... [13:10:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:10:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:13:15] (03CR) 10Joal: Add event_entity='user' to MWHistoryDeltaWriter (MERGE 6) (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290945 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [13:13:41] 06Data-Engineering, 06Data-Platform-SRE, 10EventStreams, 07Incident Severity 3, 07Wikimedia-Incident: 502/503 for mediawiki.page_change.v1 stream - https://phabricator.wikimedia.org/T427839#11976374 (10Ottomata) @tchin reminded us that eventstreams does have a memory leak when connections close {T383... [13:18:44] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to for - https://phabricator.wikimedia.org/T427553#11976402 (10Raine) [13:18:48] (03CR) 10Xcollazo: Add event_entity='user' to MWHistoryDeltaWriter (MERGE 6) (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290945 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [13:20:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:20:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:25:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:25:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:28:47] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:28:53] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:42:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: DE3.1 - Logged-out reader 21-day retention on web - https://phabricator.wikimedia.org/T424706#11976498 (10tchin) @amastilovic is there an easy way to selectively run modified dbt jobs in production for back... [13:45:24] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Add "wiki_id" to Page View Stream - https://phabricator.wikimedia.org/T427925 (10JMonton-WMF) 03NEW [14:00:47] (03CR) 10Aqu: "Looks good." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295064 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [14:08:32] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: DE3.1 - Logged-out reader 21-day retention on web - https://phabricator.wikimedia.org/T424706#11976640 (10tchin) > The mediawiki_database field is missing for page_visit events, which means we cannot calcul... [14:16:44] (03CR) 10Mforns: Add globalimagelinks table to sqoop (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295064 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [14:17:27] (03CR) 10Mforns: Add globalimagelinks table to sqoop (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295064 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [14:17:49] (03CR) 10Mforns: Add filerevision table to sqoop (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295069 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [14:18:36] RESOLVED: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:18:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:45:37] (03CR) 10Aqu: "Nice" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295069 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [15:08:06] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: `mediawiki.page_change.v1`: two schema validation errors causing events to be silently dropped by EventGate - https://phabricator.wikimedia.org/T421237#11976990 (10Ottomata) Met yesterday to discuss. Notes here: https://docs.google.com/document/d/17W... [15:12:36] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: `mediawiki.page_change.v1`: two schema validation errors causing events to be silently dropped by EventGate - https://phabricator.wikimedia.org/T421237#11977033 (10Ottomata) [15:33:42] 06Data-Engineering, 06SRE: Automate ingestion of netflow event stream - https://phabricator.wikimedia.org/T248865#11977336 (10ayounsi) [15:38:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:38:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:43:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:43:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:56:19] (03CR) 10Dr0ptp4kt: "Note on commonswiki versus other wikis." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295069 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [15:56:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: DE3.1 - Logged-out reader 21-day retention on web - https://phabricator.wikimedia.org/T424706#11977587 (10jwang) The `mediawiki.database` was added back since May 15. Between 4/29 and 5/14, the field is NUL... [15:59:28] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: `mediawiki.page_change.v1`: two schema validation errors causing events to be silently dropped by EventGate - https://phabricator.wikimedia.org/T421237#11977608 (10Ottomata) [16:01:29] (03PS3) 10Xcollazo: Add event_entity='user' to MWHistoryDeltaWriter (MERGE 6) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290945 (https://phabricator.wikimedia.org/T425729) [16:05:53] (03CR) 10Xcollazo: "@joal@wikimedia.org added all the basic `user_*` fields, including the ones needed for contributir metrics." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290945 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [16:08:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:08:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:10:33] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: `mediawiki.page_change.v1`: two schema validation errors causing events to be silently dropped by EventGate - https://phabricator.wikimedia.org/T421237#11977692 (10Ottomata) @APizzata-WMF looks like monthly reconcile failed because of this? https://a... [16:13:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:13:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:22:44] 06Data-Engineering, 06Data-Platform-SRE, 10Event-Platform: Delete topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951 (10JMonton-WMF) 03NEW [16:23:32] (03PS6) 10Xcollazo: Add DDL for mediawiki_history_incremental_v1 Iceberg table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287959 (https://phabricator.wikimedia.org/T425729) [16:23:46] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11977821 (10JMonton-WMF) [16:25:48] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [16:26:43] (03PS3) 10Dr0ptp4kt: Add globalimagelinks table to sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295064 (https://phabricator.wikimedia.org/T427532) [16:27:14] 06Data-Engineering, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), 07OKR-Work, 13Patch-For-Review: redioscope: periodically publish top clients to the data lake - https://phabricator.wikimedia.org/T424823#11977900 (10Ottomata) I updated docs! https://wikitech.wikimedia.org/wiki/Event_Plat... [16:29:19] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 06MW-Interfaces-Team, 10Event-Platform: MediaWiki DomainEvents - Create new User related DomainEvents - https://phabricator.wikimedia.org/T427817#11977933 (10Ottomata) I created this task as a way to start having that convo. I think it will have to b... [16:30:02] (03CR) 10Dr0ptp4kt: "Posted updates on this one that I hope satisfy the feedback (although please let me know if not!)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295064 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [16:40:48] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-main in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-main - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [16:43:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:43:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:46:15] (03PS5) 10Xcollazo: Daily revert detection: align with monthly DenormalizedRevisionsBuilder [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1293836 (https://phabricator.wikimedia.org/T427314) [16:48:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:48:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:02:40] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): FY-2026-2027 metrics dbt jobs failure due to exit code 143. - https://phabricator.wikimedia.org/T427222#11978142 (10Snwachukwu) a:03Snwachukwu [17:06:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11978153 (10Ottomata) > OTOH, that means if a page has 2 edits by the same user, where for one they were a bot and the other they were no... [17:09:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Add "wiki_id" to Page View Stream - https://phabricator.wikimedia.org/T427925#11978176 (10Ottomata) > Find the way of adding wiki_id into webrequest_frontend_text, probably inside the x-analytics field (This would be the recommended way)... [17:15:20] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Iceberg 1.2.1 JAR seems to clash with 1.6.1 on Spark 3.5 executor classpath - https://phabricator.wikimedia.org/T426801#11978205 (10BTullis) Could we update the title and description of this ticket, now that we know a b... [17:22:24] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Emit comprehensive mediawiki user block change information in an event stream - https://phabricator.wikimedia.org/T424685#11978213 (10Ottomata) > according to the Incremental MWH field venn-diagram,... [17:22:54] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Implement MERGE INTO writers for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T425729#11978216 (10Ottomata) > alterblocks is not in this stream, but I think there is other s... [17:51:16] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: DE3.1 - Logged-out reader 21-day retention on web - https://phabricator.wikimedia.org/T424706#11978414 (10jwang) > @amastilovic is there an easy way to selectively run modified dbt jobs in production for ba... [17:57:26] (03PS4) 10Mforns: Add globalimagelinks table to sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295064 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [17:58:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:58:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:00:23] (03PS5) 10Mforns: Add globalimagelinks table to sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295064 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [18:00:43] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Convert `control_map MAP` to a typed struct in `mediawiki_history_incremental_v1` - https://phabricator.wikimedia.org/T427862#11978477 (10xcollazo) Implementation is complete and tested (87 tests pass). B... [18:01:01] (03CR) 10Mforns: Add globalimagelinks table to sqoop (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1295064 (https://phabricator.wikimedia.org/T427532) (owner: 10Dr0ptp4kt) [18:10:27] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Iceberg 1.6.1 bug makes SELECTs fail due to vectorized read path being the default - https://phabricator.wikimedia.org/T426801#11978556 (10xcollazo) [18:13:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:13:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:23:36] RESOLVED: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:23:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:26:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:26:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:30:19] 06Data-Engineering, 07Sustainability: Move some analytics jobs to day time in Virginia - https://phabricator.wikimedia.org/T384166#11978665 (10Ottomata) > 1:128 webrequest sampled data And/or we could add Hive bucketing to webrequest? [18:31:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:31:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:32:27] 06Data-Engineering, 07Sustainability: Create a wmf.webrequest_sampled_128 Hive table - https://phabricator.wikimedia.org/T427978 (10Ottomata) 03NEW [18:32:41] 06Data-Engineering, 07Sustainability: Create a wmf.webrequest_sampled_128 Hive table - https://phabricator.wikimedia.org/T427978#11978685 (10Ottomata) Or wait for "webrequest 2.0" iceberg? @mforns? [18:33:53] 06Data-Engineering, 10MediaWiki-extensions-Wikibase-Client, 10Wikidata: Move the wbc_entity_usage table onto a dedicated DB shard - https://phabricator.wikimedia.org/T176273#11978686 (10Ahoelzl) [18:36:36] RESOLVED: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:36:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:44:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:44:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:54:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:54:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:59:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:59:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:08:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:08:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:13:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:13:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:18:16] (03PS1) 10Xcollazo: Fix MWHistorySnapshotMerger: DELETE+INSERT replaces MERGE, add page/user reconcile [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1296665 (https://phabricator.wikimedia.org/T427328) [19:21:48] 06Data-Engineering, 10Event-Platform: Replace getInvocationCount() in EventBus tests to support PHPUnit 10 - https://phabricator.wikimedia.org/T426505#11978800 (10Umherirrender) a:03Umherirrender [19:23:36] RESOLVED: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:23:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:26:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:26:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:31:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:31:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:41:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:41:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:46:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:46:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:53:04] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 06MW-Interfaces-Team, 10Event-Platform: MediaWiki DomainEvents - Create new User related DomainEvents - https://phabricator.wikimedia.org/T427817#11978902 (10BPirkle) >>! In T427817#11977933, @Ottomata wrote: > I created this task as a way to start h... [20:09:23] (03PS1) 10Xcollazo: Convert control_map from MAP to typed struct [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1296674 (https://phabricator.wikimedia.org/T427862) [20:20:13] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Convert `control_map MAP` to a typed struct in `mediawiki_history_incremental_v1` - https://phabricator.wikimedia.org/T427862#11979008 (10Ottomata) Discussed this one with Xabriel in... [22:10:48] !log Test Kitchen edge-unique experiments (poll 77688) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:10:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:11:08] !log Test Kitchen edge-unique experiments (poll 77689) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:11:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:11:29] !log Test Kitchen edge-unique experiments (poll 77690) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:11:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:11:49] !log Test Kitchen edge-unique experiments (poll 77691) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:11:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:12:09] !log Test Kitchen edge-unique experiments (poll 77692) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:12:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:12:29] !log Test Kitchen edge-unique experiments (poll 77693) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:12:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:12:49] !log Test Kitchen edge-unique experiments (poll 77694) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:12:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:13:09] !log Test Kitchen edge-unique experiments (poll 77695) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:13:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:13:29] !log Test Kitchen edge-unique experiments (poll 77696) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:13:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:13:49] !log Test Kitchen edge-unique experiments (poll 77697) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:13:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:14:10] !log Test Kitchen edge-unique experiments (poll 77698) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:14:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:14:30] !log Test Kitchen edge-unique experiments (poll 77699) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:14:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:14:50] !log Test Kitchen edge-unique experiments (poll 77700) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:14:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:15:10] !log Test Kitchen edge-unique experiments (poll 77701) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:15:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:15:30] !log Test Kitchen edge-unique experiments (poll 77702) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:15:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:15:50] !log Test Kitchen edge-unique experiments (poll 77703) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:15:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:16:10] !log Test Kitchen edge-unique experiments (poll 77704) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:16:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:16:31] !log Test Kitchen edge-unique experiments (poll 77705) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:16:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:16:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:16:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:16:51] !log Test Kitchen edge-unique experiments (poll 77706) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:16:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:17:11] !log Test Kitchen edge-unique experiments (poll 77707) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:17:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:17:31] !log Test Kitchen edge-unique experiments (poll 77708) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:17:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:17:51] !log Test Kitchen edge-unique experiments (poll 77709) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:17:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:18:11] !log Test Kitchen edge-unique experiments (poll 77710) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:18:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:18:31] !log Test Kitchen edge-unique experiments (poll 77711) - adds: none; removes: logged-out-retention-round12, attribution-research-2026-06-run, logged-out-retention-round11, we-1-8-mobile-account-menu, synth-aa-detect-hoisting-errors-1, growthexperiments-editattempt-anonwarning, share-highlight, logged-out-retention-round13, account-creation-reading-list-cta; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:18:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:26:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:26:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:56:36] FIRING: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:56:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag