[06:59:55] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9870055 (10JAllemandou) Hi Folks, I've been late in delivering this but it's landing as I write. The job transforming wikidata-xml-history for snapshot... [07:01:48] 06Data-Engineering, 13Patch-Needs-Improvement: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924#9870059 (10JAllemandou) a:05JAllemandou→03None [07:18:03] (03PS1) 10KCVelaga: Add cx_translators to sqoop job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1039975 (https://phabricator.wikimedia.org/T366869) [07:36:39] Hi moritzm - after bast1003 reimage, the SSH fingerprint on wikitech differs from the one I'm getting when trying to connect. Can you confirm it's an error on wikitech? [07:37:25] I'm actually currently updating the wikitech page, give me a minute :-) [07:38:05] Ah! thank you for that moritzm [07:38:43] done, new fingerprint at https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/bast1003.wikimedia.org [07:41:35] thanks so much moritzm :) [07:41:51] yw :-) [08:14:13] 06Data-Engineering, 10Data-Platform-SRE (2024.05.27 - 2024.06.16), 13Patch-For-Review: ISPDatabaseReader null pointer exception - https://phabricator.wikimedia.org/T365197#9870181 (10Gehel) [08:18:38] joal: would you have a few minutes and a working brain to pair on T365197? [08:18:39] T365197: ISPDatabaseReader null pointer exception - https://phabricator.wikimedia.org/T365197 [08:46:31] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9870288 (10AndrewTavis_WMDE) Thank you for the efforts here, @JAllemandou! Really great to have this back, and glad that it's worked out in a way where... [09:55:27] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9870427 (10JAllemandou) The transformation job finished, we have data (from superset): ` SELECT revision_id, revision_timestamp FROM wmf.mediawi... [09:58:38] 06Data-Engineering: Add job to create Wikidata partition to wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T363451#9870429 (10JAllemandou) →14Duplicate dup:03T364045 [09:58:46] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9870431 (10JAllemandou) [13:04:46] joal: ping [13:11:41] 10Data-Engineering (Q4 2024 April 1st - June 30th), 06Release-Engineering-Team, 07Spike: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS - https://phabricator.wikimedia.org/T360968#9870910 (10Ottomata) [13:11:49] 10Data-Engineering (Q4 2024 April 1st - June 30th), 06Release-Engineering-Team, 07Spike: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS - https://phabricator.wikimedia.org/T360968#9870911 (10Ottomata) Tagging Release Engineering for consultation. [13:11:52] 10Data-Engineering (Q4 2024 April 1st - June 30th), 06Release-Engineering-Team, 07Spike: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS - https://phabricator.wikimedia.org/T360968#9870909 (10Ottomata) @thcipriani I think I recall you or RelEng menti... [13:12:02] 10Data-Engineering (Q4 2024 April 1st - June 30th), 06Release-Engineering-Team, 07Spike: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS - https://phabricator.wikimedia.org/T360968#9870912 (10Ottomata) [13:19:27] 06Data-Engineering, 10Data-Platform-SRE (2024.05.27 - 2024.06.16), 13Patch-For-Review: ISPDatabaseReader null pointer exception - https://phabricator.wikimedia.org/T365197#9870940 (10Gehel) p:05Triage→03High [13:22:26] 10Quarry: query runs forever - https://phabricator.wikimedia.org/T366909 (10Wurgl) 03NEW [14:12:21] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#9871084 (10lbowmaker) [14:20:15] joal: last try for today... [15:36:52] 06Data-Engineering, 06Data Products, 10Metrics Platform Backlog: [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9871328 (10VirginiaPoundstone) Add status quo info for the usage scenarios as well as baseline performa... [15:37:31] 06Data-Engineering, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 15): [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9871329 (10VirginiaPoundstone) a:03phuedx [16:48:34] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting permissions for analytics-privatedata-users (with kerberos) for Mareike Heuer - https://phabricator.wikimedia.org/T364715#9871559 (10Dzahn) @lbowmaker Just making sure - this ticket needs an action from Data-Engineering to mov... [17:06:15] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services Migration🐤), 13Patch-For-Review: Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9871609 (10Ottomata) @Snwachukwu, can you 'archive' / blank(?) the migrated rep... [17:41:33] 06Data-Engineering, 13Patch-For-Review: Update the From: addresses of all email from DPE pipelines so that they use routable addresses - https://phabricator.wikimedia.org/T358675#9871725 (10Dzahn) The changes here mean that when we get informed that a systemd timer fails on cloud VPS we don't know where th... [18:15:33] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services Migration🐤), 13Patch-For-Review: Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9871762 (10Snwachukwu) Sure. I wil try to do that right away. [18:18:00] 06Data-Engineering, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 15): [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9871768 (10xcollazo) > Question for Joseph and Xabriel: How smart are Parquet... [18:29:17] 07Analytics-Data-Problem, 06Data-Platform, 06Movement-Insights: Unique devices per country spikes on wikifunctions - https://phabricator.wikimedia.org/T364872#9871808 (10Milimetric) ` select day, http_status, count(*) count_by_status from pageview_actor where year=2024 and month=4 and day in (19,26)... [18:37:25] 06Data-Engineering, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 15): [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9871845 (10Ottomata) Thanks @xcollazo! I think the question at hand is, how... [18:57:42] 06Data-Engineering, 07Voice & Tone: Rename geoeditors_blacklist_country - https://phabricator.wikimedia.org/T259804#9871862 (10Aklapper) [19:00:51] 06Data-Engineering, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 15): [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9871868 (10xcollazo) > How much will the presence of many irrelevant rows whe... [20:10:45] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting permissions for analytics-privatedata-users (with kerberos) for Mareike Heuer - https://phabricator.wikimedia.org/T364715#9872091 (10odimitrijevic) Approved [20:29:45] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting permissions for analytics-privatedata-users (with kerberos) for Mareike Heuer - https://phabricator.wikimedia.org/T364715#9872144 (10Dzahn) 05Stalled→03In progress [20:31:13] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting permissions for analytics-privatedata-users (with kerberos) for Mareike Heuer - https://phabricator.wikimedia.org/T364715#9872152 (10Dzahn) [20:33:47] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting permissions for analytics-privatedata-users (with kerberos) for Mareike Heuer - https://phabricator.wikimedia.org/T364715#9872170 (10Dzahn) Kerberos principal created. [20:45:04] 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 14): publish an announcement about dashiki migration - https://phabricator.wikimedia.org/T366755#9872198 (10VirginiaPoundstone) a:03VirginiaPoundstone [20:56:53] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting permissions for analytics-privatedata-users (with kerberos) for Mareike Heuer - https://phabricator.wikimedia.org/T364715#9872223 (10Dzahn) a:03Dzahn Mareike, you should have received 2 emails, one about changing the password for your Kerberos us... [21:56:24] 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 14): publish an announcement about dashiki migration - https://phabricator.wikimedia.org/T366755#9872361 (10VirginiaPoundstone) Message sent: [cross posting to] wikitech-l and analytics-l We're migrating Dashiki to newer... [22:01:06] 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 14): publish an announcement about dashiki migration - https://phabricator.wikimedia.org/T366755#9872369 (10VirginiaPoundstone) I managed to fudge it up and send it to analytics-l sans title. But Wikitech-l got a title. [22:08:48] 06Data-Engineering, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 15): [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9872387 (10Ottomata) > We partition the table by experiment_id. Both Iceberg... [22:24:11] 07Analytics-Data-Problem, 06Data-Platform, 06Movement-Insights: Unique devices per country spikes on wikifunctions - https://phabricator.wikimedia.org/T364872#9872508 (10Mayakp.wiki) Dan and I looked at this a bit more in our data operating theater and have a strong suspicion that this is indeed caused by re...