[02:48:06] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11208754 (10nshahquinn-wmf) @JAllemandou there's actually a bug in the new data: the domains names for Wikidata, Wikifunctions, and MediaWiki.org ar... [07:21:16] !log change the Druid public (AQS) connection string to druid1011 as we decommission druid1007 T405446 [07:21:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:21:22] T405446: Decommission druid100[7-8].eqiad.wmnet - https://phabricator.wikimedia.org/T405446 [07:27:50] (03CR) 10Brouberol: [C:03+1] "Approved as the change is sensible and nicely commented (thanks for that!). I'd probably look for a second approval, given my lack of expe" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1190705 (owner: 10Lucas Werkmeister (WMDE)) [08:41:17] 06Data-Engineering, 06cloud-services-team, 06Community-Tech, 10Data-Services, 10Multiblocks: Unexpected error "Subquery returns more than 1 row" on wiki replicas - https://phabricator.wikimedia.org/T404473#11209201 (10BTullis) All done for `an-redacteddb1001`. Thanks, all. [08:41:58] 06Data-Engineering, 06cloud-services-team, 06Community-Tech, 10Data-Services, 10Multiblocks: Unexpected error "Subquery returns more than 1 row" on wiki replicas - https://phabricator.wikimedia.org/T404473#11209202 (10BTullis) a:05BTullis→03SD0001 [08:43:18] 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Do performance testing of a big Hadoop Table hosted by Ceph - https://phabricator.wikimedia.org/T381416#11209204 (10BTullis) p:05Medium→03High [11:14:22] 06Data-Engineering, 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11209549 (10BTullis) This all feels very achievable, but I wonder if we might be making things difficult for ourselves by trying to define //one// operator that c... [11:27:35] 06Data-Engineering, 06cloud-services-team, 06Community-Tech, 10Data-Services, 10Multiblocks: Unexpected error "Subquery returns more than 1 row" on wiki replicas - https://phabricator.wikimedia.org/T404473#11209584 (10SD0001) >>! In T404473#11209201, @BTullis wrote: > All done for `an-redacteddb1001`. Th... [11:40:57] !log depool druid100[7-8] from the druid public cluster T403801 [11:41:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:41:02] T403801: decommission druid100[7-8].eqiad.wmnet - https://phabricator.wikimedia.org/T403801 [11:44:17] !log start decommissioning druid100[7-8] from the druid coordinator UI T403801 [11:44:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:51:31] 06Data-Engineering: Clean up artifacts.yaml - https://phabricator.wikimedia.org/T405379#11210081 (10Ottomata) > This might be quite onerous on ops week duty and/or folks just trying to upgrade or deploy their job. Yes, but I mean that each time someone deploys something say for Refine events, they will also be... [13:55:59] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Investigate reasons for remaining inconsistencies - https://phabricator.wikimedia.org/T385112#11210107 (10Ottomata) BTW, I looked into {T400380}, and it may be that since EventBus was switched to DomainEve... [14:14:31] 06Data-Engineering, 06cloud-services-team, 06Community-Tech, 10Data-Services, 10Multiblocks: Unexpected error "Subquery returns more than 1 row" on wiki replicas - https://phabricator.wikimedia.org/T404473#11210197 (10fnegri) @SD0001 apologies, I think I did something wrong yesterday and the change was n... [14:23:46] (03CR) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [15:14:37] (03PS6) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [15:18:19] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06MW-Interfaces-Team, 06Product-Analytics, 10Event-Platform, 13Patch-For-Review: performer struct fields NULL in event_sanitized.mediawiki_revision_tags_change - https://phabricator.wikimedia.org/T352899#11210474 (10Ottomata) a:03Ottomata [15:24:39] 06Data-Engineering, 10ChangeProp, 10WMF-JobQueue, 10Continuous-Integration-Config, and 2 others: Run EventBus tests in MediaWiki core CI - https://phabricator.wikimedia.org/T257583#11210502 (10Ottomata) Hm, I think this might be done, or done sufficiently? Maybe not for MW core itself, but there is quibble... [15:35:44] 06Data-Engineering: Clean up artifacts.yaml - https://phabricator.wikimedia.org/T405379#11210574 (10xcollazo) [15:39:23] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-rename, 10Event-Platform, 07Wikimedia-production-error: InvalidArgumentException: No page moved from 'File:Hospital sign.svg' to 'File:MUTCD D9-2.svg' with ID 1208082 could be foun... - https://phabricator.wikimedia.org/T387695#11210591 [15:41:47] 06Data-Engineering, 10ChangeProp, 10WMF-JobQueue, 10Continuous-Integration-Config, and 2 others: Run EventBus tests in MediaWiki core CI - https://phabricator.wikimedia.org/T257583#11210605 (10thcipriani) 05Open→03Resolved a:03hashar >>! In T257583#11210502, @Ottomata wrote: > Hm, I think this mi... [15:49:19] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform, 10Event-Platform: Warnings from handlePageHistoryVisibilityChangedEvent in EventBus from revision deletion - https://phabricator.wikimedia.org/T403648#11210635 (10Ottomata) a:03Ottomata [15:54:13] 06Data-Engineering, 06Data-Engineering-Radar, 10DPE Temporary Accounts, 06Product-Analytics, and 3 others: Ensure performer attributes in schemas clarify if the user is a temporary account - https://phabricator.wikimedia.org/T374940#11210657 (10Ottomata) I sort of feel we should only do this for newer stat... [15:56:34] 06Data-Engineering, 06Data-Engineering-Icebox, 10Event-Platform: Event Platform - Set Kafka headers from event data - https://phabricator.wikimedia.org/T351089#11210663 (10Ottomata) [15:57:22] 06Data-Engineering, 06Data-Engineering-Icebox, 10Event-Platform: Event Platform - Set Kafka headers from event data - https://phabricator.wikimedia.org/T351089#11210666 (10Ottomata) [15:59:45] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 06Discovery-Search, and 4 others: [EPIC] Expose rdf-streaming-updater.mutation content through EventStreams - https://phabricator.wikimedia.org/T294133#11210676 (10Ottomata) @dcausse should we resolve this task? [16:01:41] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Requesting Kerberos access for sd - https://phabricator.wikimedia.org/T405219#11210690 (10Ottomata) [16:04:10] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11210700 (10Ahoelzl) [16:06:00] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Clean up artifacts.yaml - https://phabricator.wikimedia.org/T405379#11210706 (10Ahoelzl) [16:07:28] 07Analytics-Data-Problem, 06Data-Engineering: Unique devices data has rows without any data - https://phabricator.wikimedia.org/T405430#11210713 (10Ahoelzl) @JAllemandou can you help assess and define next steps [16:08:47] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11210724 (10Dzahn) [16:10:05] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11210745 (10Dzahn) p:05Triage→03Medium [16:10:19] 06Data-Engineering, 06Data-Engineering-Radar, 06Machine-Learning-Team, 07Essential-Work: Make the revert risk predictions datasets available for analysis - https://phabricator.wikimedia.org/T388453#11210747 (10Ahoelzl) [16:13:55] 06Data-Engineering, 06Data-Engineering-Radar, 06Machine-Learning-Team, 07Essential-Work: Make the revert risk predictions datasets available for analysis - https://phabricator.wikimedia.org/T388453#11210764 (10Ottomata) FWIW, there is also now a `mediawiki.page_revert_risk_prediction_change.v1` stream and... [16:14:19] 06Data-Engineering, 06Data-Engineering-Radar, 06Machine-Learning-Team, 07Essential-Work: Make the revert risk predictions datasets available for analysis - https://phabricator.wikimedia.org/T388453#11210775 (10Ottomata) 05In progress→03Resolved a:03Ottomata Being bold and resolving the task. [16:16:10] 10Data-Engineering-Roadmap, 06Machine-Learning-Team, 06Wikimedia Enterprise, 07Epic, 10Event-Platform: [Event Platform] Implement PoC Event-Driven Data Pipeline for Revert Risk Model Scores using Event Platform Capabilities - https://phabricator.wikimedia.org/T338792#11210783 (10Ottomata) 05Open→0... [16:17:49] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Investigate reasons for remaining inconsistencies - https://phabricator.wikimedia.org/T385112#11210791 (10xcollazo) >>! In T385112#11210107, @Ottomata wrote: > BTW, I looked into {T400380}, and it may be t... [16:32:55] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Requesting Kerberos access for sd - https://phabricator.wikimedia.org/T405219#11210906 (10Dzahn) Hey @SD0001 please check your email. You should have received on with further instr... [16:34:15] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Requesting Kerberos access for sd - https://phabricator.wikimedia.org/T405219#11210925 (10Dzahn) 05Open→03In progress [16:35:16] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11210933 (10Dzahn) [16:41:02] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11210977 (10Dzahn) Hi Data Engineering, the requesting user appears to already have access to superset but with the exception of some dashboards. Tag... [16:41:59] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11210980 (10Dzahn) p:05Triage→03Medium [16:48:20] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to wmf for ericmill - https://phabricator.wikimedia.org/T404903#11211052 (10Dzahn) [16:52:28] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to wmf for ericmill - https://phabricator.wikimedia.org/T404903#11211098 (10Dzahn) tagging Data Engineering for visibility per https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#analytics-privatedata-users Data Engineering, can... [16:52:57] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to wmf for ericmill - https://phabricator.wikimedia.org/T404903#11211101 (10Dzahn) 05Open→03In progress p:05Triage→03Medium [16:53:19] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to wmf for ericmill - https://phabricator.wikimedia.org/T404903#11211105 (10Dzahn) [17:15:58] 06Data-Engineering, 06Data-Persistence, 10MediaWiki-Core-Revision-backend, 10MW-1.45-notes (1.45.0-wmf.21; 2025-09-30), and 2 others: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#11211184 (10Ladsgroup) Revert detection still works in beta: https://en.wikipedia.beta.wmcloud.org/w/index... [17:22:53] 06Data-Engineering: Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503 (10Ladsgroup) 03NEW [17:24:07] 06Data-Engineering, 06Data-Persistence, 10MediaWiki-Core-Revision-backend, 10MW-1.45-notes (1.45.0-wmf.21; 2025-09-30), and 2 others: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#11211242 (10Ladsgroup) [18:35:19] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to wmf for ericmill - https://phabricator.wikimedia.org/T404903#11211475 (10Ottomata) Perhaps https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#Access_Levels helps? I think what is needed is just "Shell account added to analytics-pri... [18:43:41] 06Data-Engineering, 06Data-Platform-SRE: Provide an access to MaxMind GeoIP in DSE K8S pods - https://phabricator.wikimedia.org/T405509 (10JAllemandou) 03NEW [19:03:48] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for ericmill - https://phabricator.wikimedia.org/T404903#11211565 (10Novem_Linguae) [19:05:50] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for ericmill - https://phabricator.wikimedia.org/T404903#11211572 (10Dzahn) Thanks @Ottomata I appreciate the recommendation and will go with the minimum level above the current access. Though there is a bit more c... [19:06:58] 06Data-Engineering, 06Discovery-Search, 06serviceops-radar, 10Event-Platform: [Event Platform] Store Flink HA metadata in Zookeeper - https://phabricator.wikimedia.org/T331283#11211576 (10Ottomata) [19:07:57] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for ericmill - https://phabricator.wikimedia.org/T404903#11211580 (10Dzahn) We still have to move the user account from "ldap_only" to "that other section" (which is actually shell access for some but not for others... [19:13:20] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for ericmill - https://phabricator.wikimedia.org/T404903#11211610 (10Ottomata) Hm, SSH access != posix shell account (perhaps this is a confusing term?). In order to have a uid to add to a group in data.yaml, they m... [19:19:36] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Core-Revision-backend, 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, and 3 others: MediaWiki\Revision\RevisionAccessException: Unable to load fresh row for rev_id: {rev_id} - https://phabricator.wikimedia.org/T400380#11211639 (10Ottoma... [19:20:39] 07Analytics-Data-Problem, 06Data-Engineering: Unique devices data has rows without any data - https://phabricator.wikimedia.org/T405430#11211654 (10JAllemandou) The "good" thing here is that the same pattern happens in the old non-iceberg unique-devices tables, it's not a newly introduced bug. I think you're r... [19:20:50] 07Analytics-Data-Problem, 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Unique devices data has rows without any data - https://phabricator.wikimedia.org/T405430#11211655 (10JAllemandou) a:03JAllemandou [19:32:29] (03PS1) 10Joal: Prevent unique-devices jobs to load empty rows [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191150 (https://phabricator.wikimedia.org/T405430) [19:33:49] 07Analytics-Data-Problem, 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 13Patch-For-Review: Unique devices data has rows without any data - https://phabricator.wikimedia.org/T405430#11211705 (10JAllemandou) @nshahquinn-wmf if you could have a look at the patch above that'd be great, thank you so... [19:37:50] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11211709 (10JAllemandou) While I understand the not-canonical concern, it happens that we've been using the non-`www` domain since the beginning of... [20:02:32] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for ericmill - https://phabricator.wikimedia.org/T404903#11211749 (10Dzahn) All that being said, @EMill-WMF , you have been upgraded from "level 1" to "level 2" in this: https://wikitech.wikim... [20:05:46] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for ericmill - https://phabricator.wikimedia.org/T404903#11211760 (10Dzahn) 05In progress→03Open a:03EMill-WMF Could you confirm if things you expected to work are working now? Thanks! [20:09:47] 06Data-Engineering, 10Event-Platform: Schema validation for EventStreamConfig - https://phabricator.wikimedia.org/T405516 (10mpopov) 03NEW [20:15:40] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for ericmill - https://phabricator.wikimedia.org/T404903#11211802 (10Novem_Linguae) FYI, I've filed {T405517}, which might be a good spot to continue that part of the discussion. [22:19:33] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Airflow jobs to do monthly XML dumps - https://phabricator.wikimedia.org/T384381#11212290 (10xcollazo) [22:34:31] (03PS7) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [22:39:05] (03CR) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. (037 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [22:46:32] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Implement the data layout, UI, and documentation for the XML file export - https://phabricator.wikimedia.org/T401022#11212343 (10Ahoelzl) a:03xcollazo [22:47:00] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Data Pipeline - https://phabricator.wikimedia.org/T405039#11212348 (10Ahoelzl) a:03amastilovic [22:47:18] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - backfill data - https://phabricator.wikimedia.org/T405040#11212350 (10Ahoelzl) a:03amastilovic [22:48:10] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11212356 (10Ahoelzl) a:03mforns [23:00:04] 07Analytics-Data-Problem, 06Data-Engineering: Unique devices data uses non-standard domains for Wikidata, Wikifunctions, and MediaWiki.org - https://phabricator.wikimedia.org/T405533 (10nshahquinn-wmf) 03NEW [23:00:52] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11212394 (10nshahquinn-wmf) @JAllemandou I filed T405533 to continue the conversation. [23:14:01] 06Data-Engineering, 06Product-Analytics, 10superset.wikimedia.org: Improve Superset's error message for draft dashboards - https://phabricator.wikimedia.org/T405535 (10nettrom_WMF) 03NEW [23:14:10] 07Analytics-Data-Problem, 06Data-Engineering: Unique devices data uses non-standard domains for Wikidata, Wikifunctions, and MediaWiki.org - https://phabricator.wikimedia.org/T405533#11212442 (10nshahquinn-wmf) >>! In T401666#11211709, @JAllemandou wrote: > While I understand the not-canonical concern, it happ...