[00:34:53] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work, 05Goal: [GOAL] Tidy up EventLogging - https://phabricator.wikimedia.org/T408059#11543743 (10Sfaci) >>! In T408059#11543345, @cjming wrote: > just to be crystal about what is being deprecated/removed from EventLoggi... [01:25:18] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [01:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [02:17:07] 06Data-Engineering: Design Schema for page state and page state with content (enriched) streams - https://phabricator.wikimedia.org/T308017#11543861 (10Ottomata) [02:34:46] 06Data-Engineering, 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision content - https://phabricator.wikimedia.org/T415158#11543886 (10Ottomata) [02:36:44] 06Data-Engineering, 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision content - https://phabricator.wikimedia.org/T415158#11543889 (10Ottomata) [02:37:15] 06Data-Engineering, 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision content - https://phabricator.wikimedia.org/T415158#11543890 (10Ottomata) We know that we will need a stable identifier for a specific rendering. Can/sho... [05:25:18] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [05:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [05:56:16] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11544028 (10Marostegui) [06:03:29] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11544032 (10Marostegui) [06:55:23] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11544087 (10Marostegui) [06:55:36] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11544091 (10Marostegui) [08:40:18] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11544194 (10Marostegui) [09:13:43] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen: Deprecate and remove mw.eventLog.submitClick() - https://phabricator.wikimedia.org/T415210#11544253 (10phuedx) [09:14:17] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitClick() - https://phabricator.wikimedia.org/T415210#11544254 (10phuedx) [09:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [09:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [10:45:26] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Deprecate and remove EventLogging::getMetricsPlatformClient() - https://phabricator.wikimedia.org/T415246 (10phuedx) 03NEW [10:46:21] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work, 05Goal: [GOAL] Tidy up EventLogging - https://phabricator.wikimedia.org/T408059#11544462 (10phuedx) >>! In T408059#11543345, @cjming wrote: > - MP methods/property in [[ https://gerrit.wikimedia.org/r/plugins/gitil... [10:47:46] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Deprecate and remove EventLogging::getMetricsPlatformClient() - https://phabricator.wikimedia.org/T415246#11544474 (10phuedx) [11:35:22] !log Test Kitchen mw-user experiment (poll 33701) - adds: none; removes: growthexperiments-revise-tone; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:35:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:58:29] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 13Patch-For-Review, and 2 others: Migrate 1 instrument using mw.eventLog.newInstrument() to mw.xLab.getInstrument() - https://phabricator.wikimedia.org/T408096#11544676 (10phuedx) [12:10:09] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 13Patch-For-Review, and 2 others: Migrate 1 instrument using mw.eventLog.newInstrument() to mw.xLab.getInstrument() - https://phabricator.wikimedia.org/T408096#11544716 (10phuedx) [12:10:16] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254 (10phuedx) 03NEW [12:10:39] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11544728 (10phuedx) [12:12:20] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11544741 (10phuedx) [13:01:54] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Deprecate and remove EventLogging::getMetricsPlatformClient() - https://phabricator.wikimedia.org/T415246#11544922 (10phuedx) [13:02:32] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work, 05Goal: [GOAL] Tidy up EventLogging - https://phabricator.wikimedia.org/T408059#11544928 (10phuedx) [13:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [13:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [13:58:47] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10MediaWiki-General: Update pingback MediaWiki versions to include new values - https://phabricator.wikimedia.org/T413349#11545093 (10xcollazo) >>! In T413349#11543338, @xcollazo wrote: > After fixes, the backfill is running well with: > > ` > airflow... [14:02:32] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10MediaWiki-General: Update pingback MediaWiki versions to include new values - https://phabricator.wikimedia.org/T413349#11545115 (10xcollazo) >>! In T413349#11543401, @cicalese wrote: > @xcollazo Thank you so much for your work on this! I appreciate i... [14:33:47] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): mw_content_history_reconcile_enrich api call returned 503 - https://phabricator.wikimedia.org/T415264 (10APizzata-WMF) 03NEW [14:36:48] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 06Data-Platform-SRE (2026.01.05 - 2026.01.23): Grant Access to analytics-privatedata-users for hmonroy - https://phabricator.wikimedia.org/T414375#11545247 (10Gehel) [14:47:32] btullis: I had no idea that Airflow made bare pods -- would it be easy to have it make Jobs instead? [14:52:32] it's possible cdanis --> https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html#kubernetesjoboperator [14:55:12] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): aggregate_for_fundraising_hourly failing for last 24 hours - https://phabricator.wikimedia.org/T415267 (10xcollazo) 03NEW [15:30:52] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Put aggregate_for_fundraising.hql into refinery - https://phabricator.wikimedia.org/T415275 (10amastilovic) 03NEW [15:34:43] (03PS1) 10Aleksandar Mastilovic: Add aggregate pageview HQL file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1230370 (https://phabricator.wikimedia.org/T415275) [15:37:10] (03CR) 10Joal: [C:03+1] Add aggregate pageview HQL file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1230370 (https://phabricator.wikimedia.org/T415275) (owner: 10Aleksandar Mastilovic) [15:38:10] !log Test Kitchen edge-unique experiments (poll 34425) - adds: none; removes: none; fields: synth-aa-test-traffic-impact - xLab/MPIC/TK tips at https://w.wiki/FwuD [15:38:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:39:12] (03CR) 10Xcollazo: [C:03+1] Add aggregate pageview HQL file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1230370 (https://phabricator.wikimedia.org/T415275) (owner: 10Aleksandar Mastilovic) [15:39:56] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Add aggregate pageview HQL file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1230370 (https://phabricator.wikimedia.org/T415275) (owner: 10Aleksandar Mastilovic) [15:43:15] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): aggregate_for_fundraising_hourly failing for last 24 hours - https://phabricator.wikimedia.org/T415267#11545679 (10xcollazo) [15:43:18] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Put aggregate_for_fundraising.hql into refinery - https://phabricator.wikimedia.org/T415275#11545677 (10xcollazo) →14Duplicate dup:03T415267 [15:45:00] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): aggregate_for_fundraising_hourly failing for last 24 hours - https://phabricator.wikimedia.org/T415267#11545686 (10xcollazo) Related patch from @amastilovic: https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1230370 [15:50:03] 06Data-Engineering, 06Reader Growth Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog, and 3 others: Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11545711 (10Jgiannelos) This should be live after the last d... [15:57:18] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): aggregate_for_fundraising_hourly failing for last 24 hours - https://phabricator.wikimedia.org/T415267#11545737 (10amastilovic) a:03amastilovic [16:08:30] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Add user ebysans and amastilovic platform-eng airflow instance admins - https://phabricator.wikimedia.org/T414353#11545803 (10xcollazo) [16:08:48] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Add user ebysans and amastilovic platform-eng airflow instance admins - https://phabricator.wikimedia.org/T414353#11545807 (10xcollazo) (added @amastilovic on this ticket as well) [16:17:58] 06Data-Engineering, 10MediaWiki-General: Refactor pingback analytics pipeline - https://phabricator.wikimedia.org/T415283 (10cicalese) 03NEW [16:18:39] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10MediaWiki-General: Update pingback MediaWiki versions to include new values - https://phabricator.wikimedia.org/T413349#11545847 (10cicalese) Done! See T415283. [16:33:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Add user ebysans and amastilovic platform-eng airflow instance admins - https://phabricator.wikimedia.org/T414353#11545931 (10xcollazo) 05Open→03Resolved This got solved in a [[... [17:13:30] 06Data-Engineering, 10AQS2.0: Introduce a new AQS endpoint to expose video plays - https://phabricator.wikimedia.org/T415202#11546097 (10Ahoelzl) @Ladsgroup we need to understand priority and scope. @GGoncalves-WMF will reach out to you. [17:16:44] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11546111 (10Milimetric) need DE here? Please re-add if so [17:18:19] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Deprecate and remove EventLogging::getMetricsPlatformClient() - https://phabricator.wikimedia.org/T415246#11546118 (10Milimetric) watching, please add us if you need review / brainbouncing [17:18:31] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitClick() - https://phabricator.wikimedia.org/T415210#11546133 (10Milimetric) [17:18:44] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Technical-Debt: Deprecate and remove mw.eventLog.submitClick() - https://phabricator.wikimedia.org/T415210#11546140 (10Milimetric) watching, please add us if you need review / brainbouncing [17:19:24] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Secret management on airflow for the automated transfer of (public) datasets from stats infra --> WME AWS - https://phabricator.wikimedia.org/T415208#11546149 (10Ahoelzl) [17:21:09] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Datasets-General-or-Unknown: Get dump mirrors to use new dumps-rsync service name - https://phabricator.wikimedia.org/T415193#11546153 (10Ahoelzl) [17:22:00] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision content - https://phabricator.wikimedia.org/T415158#11546162 (10Ahoelzl) [17:23:02] 06Data-Engineering: Backfill `user_central_id` on wmf_content.mediawiki_content_* tables - https://phabricator.wikimedia.org/T414832#11546173 (10Milimetric) as the backfill is not strictly needed now we will keep it in mind [17:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [17:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [17:25:41] 06Data-Engineering, 07Essential-Work: Add robots.txt to dumps.wikimedia.org - https://phabricator.wikimedia.org/T408954#11546197 (10Ahoelzl) a:05Ahoelzl→03None [17:26:09] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11546208 (10Ahoelzl) [17:26:32] 06Data-Engineering, 07Essential-Work, 06Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)): [Renaming TestKitchen] Update custom-data-monitor - https://phabricator.wikimedia.org/T414451#11546212 (10Milimetric) @Sfaci are you going to make the changes? [17:26:40] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 06Test Kitchen, 07Essential-Work: Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11546215 (10Ahoelzl) [17:30:02] 06Data-Engineering: Manage druid `webrequest_sampled_live` data size - https://phabricator.wikimedia.org/T398236#11546236 (10Ahoelzl) This should be discussed in the context of the dedicated SRE Druid cluster work. [17:33:25] !log Test Kitchen mw-user experiment (poll 34768) - adds: growthexperiments-revise-tone; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [17:33:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:02:20] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Publish Dumps 2 to dumps.wikimedia.org and provide only monthly dumps - https://phabricator.wikimedia.org/T414389#11546398 (10xcollazo) >>! In T414389#11540482, @Poslovitch wrote: > Hi, is this why the mid-month dump run (20260120) has not started? Yes.... [18:14:50] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Change delete selection for SLO metric - https://phabricator.wikimedia.org/T414779#11546473 (10xcollazo) [18:16:31] 06Data-Engineering, 10MediaWiki-General: Refactor pingback analytics pipeline - https://phabricator.wikimedia.org/T415283#11546478 (10xcollazo) Copying the issues found on the original pipeline here, for completeness: >>! In T413349#11543337, @xcollazo wrote: > We ran into a couple issues trying to backfill:... [18:26:57] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Refine generates very large XCOM values - https://phabricator.wikimedia.org/T414953#11546523 (10Antoine_Quhen) a:03Antoine_Quhen [20:58:52] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06MW-Interfaces-Team, 10Event-Platform: Common event data model for data derived from parsed page revision content - https://phabricator.wikimedia.org/T415158#11546797 (10Ottomata) I've been doing a little brainstorming. H... [21:25:19] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest_sampled ingested an unexpected number of records for a Kafka topic partition. ... [21:25:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest_sampled&var-kafka_topic=webrequest_sampled&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [22:21:03] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07OKR-Work, 13Patch-For-Review: Run dbt from Airflow - https://phabricator.wikimedia.org/T410268#11546998 (10Ahoelzl) [22:25:32] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add data-steward-alerts to pageview_human_bot_daily DAG - https://phabricator.wikimedia.org/T415316 (10Ahoelzl) 03NEW [22:59:38] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06Product Safety and Integrity: FYI: Changes to the cuc_agent column in the cu_changes table - https://phabricator.wikimedia.org/T361210#11547123 (10Dreamy_Jazz) It is now possible to use the `cuc_agent_id` column and `cu_useragent` table to read t...