[01:39:07] (03PS1) 10Xcollazo: Drop _within_90_days revert tier; full-table revert seed; scoped back-patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1293836 (https://phabricator.wikimedia.org/T427314) [01:45:22] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Spike: drop all 90-day windows in MWHistoryDeltaWriter and replace with full-table revert detection - https://phabricator.wikimedia.org/T427314#11957788 (10xcollazo) **Spike complete — full-table r... [01:54:30] (03CR) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [02:08:00] FIRING: [3x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:08:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:08:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [02:13:00] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:13:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:13:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [02:23:01] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:23:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:23:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [02:28:00] RESOLVED: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:28:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:28:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [03:38:46] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list May 2026 - https://phabricator.wikimedia.org/T427343 (10GFontenelle_WMF) 03NEW [03:42:40] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Pageviews-API: PageView tool not updated since 21 May 2026 - https://phabricator.wikimedia.org/T427171#11957892 (10Robertsky) Looks good to me. [05:08:51] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11957917 (10AKhatun_WMF) [05:16:06] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11957920 (10AKhatun_WMF) [05:27:59] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11957927 (10AKhatun_WMF) [05:52:52] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#11957948 (10AKhatun_WMF) Full monthly load stats for all wikis, `2026-03` snapshot. Time taken: ~6 min and ~11min respectively. Total ~17 minutes. ` dbt run -... [07:11:29] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#11958008 (10elukey) Trying to summarize the work to do by layers: haproxy: here https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/106 should be e... [07:37:13] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#11958059 (10JAllemandou) > I think the only rule we'd have to follow afterwards is being sure to adding new keys to the struct before we start emitting them into kaf... [07:40:38] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#11958061 (10JAllemandou) > haproxy: here https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/106 should be enough, like we do with X-Analytics - the... [07:43:15] 06Data-Engineering: Create an official `webrequest_sampled` table in the datalake - https://phabricator.wikimedia.org/T427347 (10JAllemandou) 03NEW [07:55:05] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#11958158 (10elukey) >>! In T427068#11958061, @JAllemandou wrote: >> haproxy: here https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/106 should be... [08:44:14] !log Test Kitchen edge-unique experiments (poll 49537) - adds: synth-aa-detect-hoisting-errors-1; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [08:44:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:24:02] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#11958828 (10Fabfur) So IIUC in this case we have 2 possibilities: 1. Use a new header/variable (x-provenance) that needs to be logged into haproxy standard log form... [10:29:16] (03PS5) 10Seanleong-wmde: Script to gather metrics for Recent Changes in pilot Wikis. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1288258 (https://phabricator.wikimedia.org/T426384) [10:31:03] (03PS6) 10Seanleong-wmde: Script to gather metrics for Recent Changes in pilot Wikis. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1288258 (https://phabricator.wikimedia.org/T426384) [10:57:04] (03CR) 10Neslihan Turan: [C:03+1] "Looked good to me but I am not familiar with this scripts, so someone else should see it." [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1288258 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [10:59:00] (03CR) 10Nicholusmuwonge: Script to gather metrics for Recent Changes in pilot Wikis. (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1288258 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [11:07:10] (03CR) 10Nicholusmuwonge: Script to gather metrics for Recent Changes in pilot Wikis. (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1288258 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [11:41:11] (03CR) 10Joal: Drop _within_90_days revert tier; full-table revert seed; scoped back-patch (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1293836 (https://phabricator.wikimedia.org/T427314) (owner: 10Xcollazo) [12:34:30] !log Test Kitchen edge-unique experiments (poll 50223) - adds: logged-out-retention-round13; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [12:34:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:25:56] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list May 2026 - https://phabricator.wikimedia.org/T427343#11959555 (10mforns) It's merged, moving this to done :-) [13:26:23] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list May 2026 - https://phabricator.wikimedia.org/T427343#11959562 (10mforns) [13:41:57] !log Test Kitchen edge-unique experiments (poll 50424) - adds: none; removes: none; fields: synth-aa-detect-hoisting-errors-1 - xLab/MPIC/TK tips at https://w.wiki/FwuD [13:41:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:09:45] (03CR) 10Milimetric: [V:03+2 C:03+2] "merging since no objections and this needs to go live soon." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1276803 (https://phabricator.wikimedia.org/T417050) (owner: 10Milimetric) [14:42:40] (03PS1) 10Joal: Update changelog.md for new release v0.3.15 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1294309 [14:44:58] (03CR) 10Joal: [V:03+2 C:03+2] "Self merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1294309 (owner: 10Joal) [15:04:42] 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10MediaWiki-extensions-WikimediaEvents, and 4 others: Could not hoist data into experiment.subject_id for event - https://phabricator.wikimedia.org/T421152#11959989 (10KReid-WMF) 05Open→03Resolved [15:18:36] (03PS2) 10Xcollazo: Drop _within_90_days revert tier; full-table revert seed; scoped back-patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1293836 (https://phabricator.wikimedia.org/T427314) [15:19:51] (03CR) 10Xcollazo: Drop _within_90_days revert tier; full-table revert seed; scoped back-patch (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1293836 (https://phabricator.wikimedia.org/T427314) (owner: 10Xcollazo) [15:26:54] (03CR) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [15:31:54] (03CR) 10CI reject: [V:04-1] Drop _within_90_days revert tier; full-table revert seed; scoped back-patch [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1293836 (https://phabricator.wikimedia.org/T427314) (owner: 10Xcollazo) [15:40:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Engineering-Radar, 06ServiceOps new, 10ServiceOps-Services-Oids, and 3 others: Make eventstreams-internal available to WMF staff without an ssh tunnel - https://phabricator.wikimedia.org/T348763#11960132 (10atsuko) dse-k8s service applied and wo... [15:59:55] 06Data-Engineering, 06Machine-Learning-Team: EventStreams - Expose mediawiki.page_outlink_topic_prediction_change.v1 stream - https://phabricator.wikimedia.org/T427416 (10Ottomata) 03NEW [16:14:46] (03PS13) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) [16:16:49] (03CR) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [16:53:51] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10PageViewInfo, 06ServiceOps new, 10ServiceOps-SharedInfra: Migrate PageViewInfo calls away from rest-gateway - https://phabricator.wikimedia.org/T411771#11960574 (10Ahoelzl) [16:57:45] 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Platform, 06Growth-Team, and 5 others: Image Suggestions uses AI-generated images from Commons when adding images on English Wikipedia - https://phabricator.wikimedia.org/T422513#11960595 (10SherryYang-WMF) [17:24:46] (03CR) 10Xcollazo: "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1293836 (https://phabricator.wikimedia.org/T427314) (owner: 10Xcollazo) [17:41:16] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Engineering-Radar, 06ServiceOps new, 10ServiceOps-Services-Oids, and 2 others: Make eventstreams-internal available to WMF staff without an ssh tunnel - https://phabricator.wikimedia.org/T348763#11960788 (10atsuko) Cleanup: 0. Check monitorings... [19:29:35] 06Data-Engineering: Delete `wmf_raw.mediawiki_content` table after a few months - https://phabricator.wikimedia.org/T427441 (10JAllemandou) 03NEW [19:30:05] 06Data-Engineering: Delete `wmf_raw.mediawiki_content` table after a few months - https://phabricator.wikimedia.org/T427441#11961061 (10JAllemandou) [19:30:07] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11961062 (10JAllemandou) [22:10:46] (03PS1) 10Milimetric: Add one-off tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1294434 [22:43:57] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457 (10Ahoelzl) 03NEW [22:44:39] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#11961482 (10Ahoelzl) [22:44:40] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Load Google Search Console data into the Data Lake - https://phabricator.wikimedia.org/T420996#11961483 (10Ahoelzl)