[03:04:17] 06Data-Engineering: [Spike] Comparitive game stats - https://phabricator.wikimedia.org/T405865 (10Seddon) 03NEW [03:05:02] 06Data-Engineering, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: [Spike] Comparitive game stats - https://phabricator.wikimedia.org/T405865#11223114 (10Seddon) [03:05:06] 06Data-Engineering, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: [Spike] Comparitive game stats - https://phabricator.wikimedia.org/T405865#11223116 (10Seddon) [04:35:02] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11223140 (10achou) Based on the discussion wit... [07:42:22] (03PS1) 10Aqu: WIP: Script to manually add a map-column to mediawiki_content_history_v1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191982 (https://phabricator.wikimedia.org/T388793) [07:47:14] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11223317 (10Krinkle) [07:50:47] (03CR) 10CI reject: [V:04-1] WIP: Script to manually add a map-column to mediawiki_content_history_v1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191982 (https://phabricator.wikimedia.org/T388793) (owner: 10Aqu) [07:54:42] (03PS2) 10Aqu: WIP: Script to manually add a map-column to mediawiki_content_history_v1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191982 (https://phabricator.wikimedia.org/T388793) [09:12:14] (03CR) 10Aqu: [V:03+1 C:03+1] "Looks good." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191150 (https://phabricator.wikimedia.org/T405430) (owner: 10Joal) [12:30:19] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for later deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191150 (https://phabricator.wikimedia.org/T405430) (owner: 10Joal) [13:11:06] 06Data-Engineering, 06Infrastructure-Foundations: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11224492 (10Ottomata) @brouberol it would be useful to mirror all Event Platform compatible topics, e.g. those with streams declared in EventStream... [13:19:09] (03CR) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [14:10:20] 06Data-Engineering, 06Infrastructure-Foundations: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11224786 (10brouberol) Cool, I can inject the topics via a script we could commit to `deployment-charts` to maintain that topic list over time: `la... [14:12:00] (03CR) 10Joal: "One nit in comment, otherwise looks good to me" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 (owner: 10Snwachukwu) [14:14:57] (03CR) 10Ottomata: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. (038 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [14:21:52] (03CR) 10Ottomata: "I know this is for sqoop, but is there are reason we want to couple this with sqoop naming? Could we be just a little bit more future pro" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 (owner: 10Snwachukwu) [14:28:52] (03PS6) 10Snwachukwu: CREATE HQL SCRIPT TO UPDATE SCOOP WIKI LIST DATA FILE. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 [14:43:24] 06Data-Engineering, 06Infrastructure-Foundations: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11224921 (10brouberol) I had a look in grafana, and the overall size of the topics is very small. {F66710393} Now that we have a k8s cluster in c... [14:48:43] 06Data-Engineering: Design Schema for page state and page state with content (enriched) streams - https://phabricator.wikimedia.org/T308017#11224953 (10Ottomata) [14:48:46] 06Data-Engineering, 06tech-decision-forum, 10Event-Platform: MediaWiki Event Carried State Transfer - Problem Statement - https://phabricator.wikimedia.org/T291120#11224952 (10Ottomata) [14:53:32] 06Data-Engineering, 10AQS2.0, 07Documentation: Adding a AQS 2.0 endpoint guide - https://phabricator.wikimedia.org/T356748#11224984 (10Ottomata) We are about to add an AQS endpoint for {T405041}. Could the linked google doc please be shared? Thank you! [14:58:42] 06Data-Engineering, 06Traffic, 13Patch-For-Review: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11225004 (10Ottomata) > wmfuniq_freq drive by nit: if you are storing percentages, why not just store the actual percentage rather than the percentage / 10? E.g. v... [15:06:12] !log stop and disable druid services on druid100[7-8] T403801 [15:06:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:06:17] T403801: decommission druid100[7-8].eqiad.wmnet - https://phabricator.wikimedia.org/T403801 [15:19:52] (03PS7) 10Snwachukwu: CREATE HQL SCRIPT TO UPDATE SCOOP WIKI LIST DATA FILE. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 [15:20:26] (03CR) 10Snwachukwu: "This is good. I just tweaked it a bit." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 (owner: 10Snwachukwu) [15:33:21] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#11225144 (10Ottomata) Nice! Thank you. https://gerrit.wikimedia.o... [15:37:08] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10service-utils, 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Migrate and re-deploy eventgate-wikimedia using new service-utils - https://phabricator.wikimedia.org/T403169#11225157 (10Ottomata) Hm, okay! Ind... [15:39:35] 06Data-Engineering, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: [Spike] Comparitive game stats - https://phabricator.wikimedia.org/T405865#11225167 (10Ottomata) Step 1: instrument and send events that you can use to compute these metrics. Step 2: ???? ;) [15:51:02] 06Data-Engineering, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: [Spike] Comparitive game stats - https://phabricator.wikimedia.org/T405865#11225257 (10Milimetric) Step 2: define the queries more precisely Step 3: make a stream job process events and store them into the correct data store (i... [15:58:09] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#11225327 (10dcausse) >>! In T376026#11225144, @Ottomata wrote: > Nice... [16:01:06] 06Data-Engineering, 06Infrastructure-Foundations: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11225342 (10Ottomata) > That being said, Mirrormaker prefixes mirrored topics with the cluster source name This is MirrorMaker 2, I suppose, ya? S... [16:20:32] (03CR) 10Ottomata: WIP: Script to manually add a map-column to mediawiki_content_history_v1 (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191982 (https://phabricator.wikimedia.org/T388793) (owner: 10Aqu) [16:24:24] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it - https://phabricator.wikimedia.org/T388793#11225568 (10Ottomata) @Antoine_Quhen @xcollazo If we are about to do a big migration ([... [16:32:51] (03CR) 10Ottomata: CREATE HQL SCRIPT TO UPDATE SCOOP WIKI LIST DATA FILE. (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 (owner: 10Snwachukwu) [16:38:38] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Engineering-Icebox, 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables - https://phabricator.wikimedia.org/T307040#11225660 (10Ottomata) [16:38:52] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables - https://phabricator.wikimedia.org/T307040#11225661 (10Ottomata) [16:39:11] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables - https://phabricator.wikimedia.org/T307040#11225662 (10Ottomata) a:03Ottomata [16:40:12] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it - https://phabricator.wikimedia.org/T388793#11225667 (10xcollazo) Some notes from a meeting with @Antoine_Quhen: We confirmed that... [16:40:55] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables and into DataHub - https://phabricator.wikimedia.org/T307040#11225671 (10Ottomata) [16:42:47] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Rewrite wmf_content.mediawiki_content_v1 with a new column for origin_rev_id - https://phabricator.wikimedia.org/T405944 (10xcollazo) 03NEW [16:46:40] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it - https://phabricator.wikimedia.org/T388793#11225722 (10xcollazo) >>! In T388793#11225568, @Ottomata wrote: > @Antoine_Quhen @xcoll... [16:52:37] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform: Deploy mediawiki-event-enrichment Flink jobs running 1.20 - https://phabricator.wikimedia.org/T401725#11225791 (10Ottomata) [16:55:56] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#11225817 (10dcausse) >>! In T376026#11225327, @dcausse wrote: >>>! In... [16:56:28] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Observability-Logging, 10Event-Platform, 13Patch-For-Review: eventgate logs field explosion - https://phabricator.wikimedia.org/T343342#11225821 (10tchin) [16:56:31] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10service-utils, 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Migrate and re-deploy eventgate-wikimedia using new service-utils - https://phabricator.wikimedia.org/T403169#11225822 (10tchin) [17:11:21] 06Data-Engineering, 10Wikidata-Query-Service, 10Event-Platform: WDQS sparql query event generator should not set meta.dt - https://phabricator.wikimedia.org/T405949#11225903 (10Ottomata) [17:12:38] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#11225909 (10Ottomata) Thank you! [17:41:52] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11225980 (10Ottomata) [17:49:26] 06Data-Engineering, 10Observability-Alerting, 10Event-Platform: EventgateProduceRateStop alert should be active datacenter aware - https://phabricator.wikimedia.org/T405952 (10Ottomata) 03NEW [17:56:05] 06Data-Engineering, 06Infrastructure-Foundations: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11226023 (10brouberol) Ok so in that case I should //disable// topic prefixing. I'll see how I can do that. Oh and yes, this is mirrormaker2. [17:59:31] 06Data-Engineering, 06Infrastructure-Foundations: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11226027 (10brouberol) According to https://stackoverflow.com/a/70994290, I can do this by setting ` replication.policy.class=org.apache.kafka.conn... [18:00:11] 07Analytics-Data-Problem, 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Unique devices data has rows without any data - https://phabricator.wikimedia.org/T405430#11226028 (10nshahquinn-wmf) @JAllemandou I know you've already merged the task, but FWIW, I just took a look and it makes sense to me! T... [18:27:41] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Adapt MW Content pipelines to the removal of upstream revision.rev_sha1 - https://phabricator.wikimedia.org/T405641#11226094 (10xcollazo) Discussed this with @JAllemandou. We came to these conclusions: * It seems the work fro... [18:47:46] 06Data-Engineering, 06Infrastructure-Foundations: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11226191 (10Ottomata) Great, thank you! And sorry about that. One magical day in the future we'll fix the flaw and do the cluster name prefixing... [18:48:49] (03PS9) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [19:00:41] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Adapt MW Content pipelines to the removal of upstream revision.rev_sha1 - https://phabricator.wikimedia.org/T405641#11226280 (10Ottomata) > This may require changes to page_change to honor there as well? [[ https://gerrit.wiki... [19:05:36] (03CR) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [19:11:33] (03CR) 10Ottomata: [C:03+1] "Nice thank you!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [19:13:56] 10Analytics-Canonical-Data: Fetch base wiki data from SiteMatrix or similar - https://phabricator.wikimedia.org/T405960 (10nshahquinn-wmf) 03NEW p:05Triage→03Low [19:17:09] (03CR) 10Ottomata: "Oh, wait there are still a few unresolved comments, about hardcoded values / parameters." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [19:26:46] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Adapt MW Content pipelines to the removal of upstream revision.rev_sha1 - https://phabricator.wikimedia.org/T405641#11226440 (10xcollazo) > If needed, I wonder if the reconciliation / content enrichment pipelines could enrich t... [19:32:32] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11226509 (10Ahoelzl) [19:32:59] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11226515 (10Ahoelzl) p:05Triage→03High a:03xcollazo [19:35:47] (03PS10) 10Snwachukwu: Update project namespace map fields. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [19:36:48] (03CR) 10Snwachukwu: "thank you. Updated commit message!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [19:37:30] (03CR) 10Snwachukwu: [C:03+2] Update project namespace map fields. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [20:20:48] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11226779 (10Ottomata) [20:47:49] (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [20:53:02] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11226901 (10mforns) [21:30:16] 06Data-Engineering, 10AQS2.0, 07Documentation: Adding a AQS 2.0 endpoint guide - https://phabricator.wikimedia.org/T356748#11226997 (10Sfaci) @Ottomata the google doc is already shared with everybody with Editor permissions. You and others should be able to edit it. [21:36:00] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11227024 (10Eevans) >>! In T401021#11223140, @... [21:41:25] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10MediaWiki-Page-derived-data, and 2 others: Normalize categorylinks table - https://phabricator.wikimedia.org/T299951#11227062 (10Zabe) a:05Ladsgroup→03Zabe [22:04:57] 06Data-Engineering, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Provide an access to MaxMind GeoIP in DSE K8S pods - https://phabricator.wikimedia.org/T405509#11227220 (10BTullis) p:05Triage→03High [22:09:43] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11227233 (10mforns) [22:12:15] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11227243 (10mforns)