[05:57:48] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11239439 (10mforns) [06:01:40] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11239442 (10mforns) [06:04:15] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11239443 (10mforns) [06:10:31] Heya, [06:11:50] I've been told that the Wp-zero-data-bot account is still using my email address (which leads to two accounts with the same email address [06:12:38] Is the "Wp-zero-data-bot" account still used anywhere in analytics (then I'll simple change the email address in idm to anything of your choice). Otherwise I'd try to delete the account. [06:13:55] On second though ... I'll file a Phabricator ticket to have it better documented :-) Sorry for the noise [06:14:16] s/though/thought/ [06:27:30] 06Data-Engineering: Clarify what to do with the 'Wp-zero-data-bot' account - https://phabricator.wikimedia.org/T406298 (10QChris) 03NEW [06:33:38] 06Data-Engineering: Clarify what to do with the 'Wp-zero-data-bot' account - https://phabricator.wikimedia.org/T406298#11239466 (10QChris) [06:54:20] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11239483 (10mforns) [07:14:41] FIRING: MediawikiPageContentChangeEnrichAvailability: ... [07:14:41] Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability [07:19:41] RESOLVED: MediawikiPageContentChangeEnrichAvailability: ... [07:19:41] Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability [07:56:38] 06Data-Engineering-Radar, 10Research-engineering, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work, 13Patch-For-Review: Add analytics-research user to stat boxes - https://phabricator.wikimedia.org/T403207#11239608 (10Stevemunene) created the keytabs for the stat hosts and added them ` s... [08:10:06] 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 3 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#11239703 (10Gehel) 05In progress→03Resolved [08:11:32] 06Data-Engineering, 06Java-Scala-Standardization, 06Discovery-Search (2025.09.26 - 2025.10.17), 07Essential-Work: Create Gitlab CI templates for JVM packages - https://phabricator.wikimedia.org/T386406#11239716 (10Gehel) 05Open→03Resolved [09:05:29] 06Data-Engineering-Radar, 10Research-engineering, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work, 13Patch-For-Review: Add analytics-research user to stat boxes - https://phabricator.wikimedia.org/T403207#11239838 (10Stevemunene) analytics-research user has been added to stat hosts, @fka... [09:16:43] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11239865 (10JAllemandou) >>! In T406069#11238368, @Ottomata wrote: > Ah but @JAllemandou what do we... [11:54:18] (03CR) 10Gkyziridis: "Hey, @aotto@wikimedia.org could you please check this patch whenever you have time in order to merge it?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [12:04:25] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11240751 (10mforns) [12:49:37] 10Data-Engineering-Roadmap, 06Product Safety and Integrity, 06Product-Analytics, 10Temporary accounts, 07Epic: [Epic] Update schemas and instrumentation code for temporary accounts - https://phabricator.wikimedia.org/T374942#11240873 (10OKryva-WMF) [12:54:29] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 10Event-Platform: Flink images should be build on top of openjdk - https://phabricator.wikimedia.org/T400296#11240907 (10brouberol) a:05brouberol→03None [12:59:19] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11240913 (10dcausse) @brouberol thanks! This image seems to w... [13:00:40] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11240915 (10brouberol) 05In progress→03Resolved [13:00:44] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11240918 (10brouberol) Thanks! Closing this one. [13:31:12] 06Data-Engineering-Radar, 10Research-engineering, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work, 13Patch-For-Review: Add analytics-research user to stat boxes - https://phabricator.wikimedia.org/T403207#11241090 (10fkaelin) Thank you for the update. I think /etc/sudoers.d/ also has to... [14:09:14] 06Data-Engineering-Radar, 10Research-engineering, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work, 13Patch-For-Review: Add analytics-research user to stat boxes - https://phabricator.wikimedia.org/T403207#11241251 (10Stevemunene) Thanks @fkaelin this has been updated. ` stevemunene@cumi... [14:20:57] 06Data-Engineering, 06Data-Persistence, 10GlobalBlocking, 06Product Safety and Integrity, and 3 others: globalblocks table: SQL in extension and production have different type for gb_address - https://phabricator.wikimedia.org/T395669#11241297 (10OKryva-WMF) [14:31:04] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11241341 (10Dbrant) >>! In T403660#112... [14:46:27] 06Data-Engineering-Radar, 10Research-engineering, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Add analytics-research user to stat boxes - https://phabricator.wikimedia.org/T403207#11241382 (10fkaelin) Nice, this works now. Thank you. [14:48:56] 06Data-Engineering: Spike: figure out how to efficiently send XComs to Airflow dynamically mapped tasks - https://phabricator.wikimedia.org/T406341 (10xcollazo) 03NEW [14:59:30] 06Data-Engineering: Spike: figure out how to efficiently send XComs to Airflow dynamically mapped tasks - https://phabricator.wikimedia.org/T406341#11241457 (10xcollazo) [15:24:21] 06Data-Engineering, 06Data-Engineering-Radar, 10Citoid, 06MediaWiki-Engineering, and 7 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#11241631 (10Dreamy_Jazz) [15:25:04] (03PS1) 10Snwachukwu: Add check for wikis count to Mediawiki history data quality checks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1193440 (https://phabricator.wikimedia.org/T365203) [15:28:46] 06Data-Engineering, 06Data-Engineering-Radar, 06Traffic, 13Patch-For-Review: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11241706 (10CDanis) Will deploy Puppet patches on Monday. [15:32:03] 06Data-Engineering, 06Infrastructure-Foundations, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 13Patch-For-Review: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11241729 (10CDanis) @brouberol Let's deploy together Monday, my morning/your... [15:41:28] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11241782 (10mforns) [15:46:03] (03CR) 10TChin: Add check for wikis count to Mediawiki history data quality checks (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1193440 (https://phabricator.wikimedia.org/T365203) (owner: 10Snwachukwu) [15:56:08] 06Data-Engineering, 07OKR-Work: Work on client-side Bot Detection - https://phabricator.wikimedia.org/T406359#11242005 (10Milimetric) [15:56:12] 06Data-Engineering, 07OKR-Work: Work on client-side Bot Detection - https://phabricator.wikimedia.org/T406359#11242010 (10Milimetric) p:05Triage→03High a:03Milimetric [15:56:34] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242018 (10mforns) [15:59:46] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 10Temporary accounts, and 2 others: Prepare EventLogging for temp accounts - https://phabricator.wikimedia.org/T374812#11242065 (10OKryva-WMF) [16:00:40] (03PS2) 10Snwachukwu: Add check for wikis count to Mediawiki history data quality checks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1193440 (https://phabricator.wikimedia.org/T365203) [16:06:31] (03PS3) 10Snwachukwu: Add check for wikis count to Mediawiki history data quality checks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1193440 (https://phabricator.wikimedia.org/T365203) [16:06:50] (03CR) 10Snwachukwu: "Please check my recent patch." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1193440 (https://phabricator.wikimedia.org/T365203) (owner: 10Snwachukwu) [16:13:44] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06Trust and Safety Product Team, and 2 others: Add '*_actor_ip_hex_time' indexes to 'cu_changes', 'cu_log_event', and 'cu_private_event' - https://phabricator.wikimedia.org/T395683#11242173 (10OKryva-WMF) [16:28:18] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Airflow jobs to do monthly XML dumps - https://phabricator.wikimedia.org/T384381#11242210 (10xcollazo) Ran the following as `hdfs`: ` $ whoami hdfs $ hostname -f an-launcher1002.eqiad.wmnet $ hdfs dfs -m... [16:46:05] !log ran a bunch of hdfs dfs commands as the hdfs user to setup /wmf/data/exports. Details at T384381#11242210. [16:46:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:46:09] T384381: Airflow jobs to do monthly XML dumps - https://phabricator.wikimedia.org/T384381 [17:05:46] (03CR) 10TChin: [C:03+2] Add check for wikis count to Mediawiki history data quality checks (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1193440 (https://phabricator.wikimedia.org/T365203) (owner: 10Snwachukwu) [17:16:33] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242360 (10mforns) [17:19:54] (03Merged) 10jenkins-bot: Add check for wikis count to Mediawiki history data quality checks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1193440 (https://phabricator.wikimedia.org/T365203) (owner: 10Snwachukwu) [17:20:31] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242366 (10mforns) [17:24:38] 06Data-Engineering, 06Experimentation Lab, 10MediaWiki-extensions-WikimediaEvents, 13Patch-For-Review: Prepare mediawiki-client-error Logstash dashboards for mobile subdomain sunsetting - https://phabricator.wikimedia.org/T400852#11242373 (10Jdlrobson-WMF) 05Stalled→03Open I think this is only stalled... [18:28:28] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps - https://phabricator.wikimedia.org/T366752#11242510 (10xcollazo) [18:36:19] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242542 (10mforns) [18:41:25] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242550 (10mforns) [18:47:42] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Rewrite wmf_content.mediawiki_content_v1 with a new column for origin_rev_id - https://phabricator.wikimedia.org/T405944#11242556 (10xcollazo) [18:53:33] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242560 (10mforns) [18:55:01] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: Rewrite wmf_content.mediawiki_content_v1 with a new column for origin_rev_id - https://phabricator.wikimedia.org/T405944#11242574 (10xcollazo) Another wrench we discussed: It would be quite involved to modify the reconcile alg... [18:57:21] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242590 (10mforns) [19:33:42] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11242738 (10BTracy-WMF) 05Resolved→03Open When trying to access https://superset.wikimedia.org, I'm receiving the error message "Service access denied d... [19:41:44] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11242771 (10Dzahn) Yes, it seems like that is the case. This goes back to T405366#11210719. I think you also needed to request the LDAP group "wmf". Which... [19:44:35] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11242774 (10BTracy-WMF) 05Open→03Resolved I have access now, thanks @Dzahn ! [19:44:41] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11242776 (10Dzahn) 05Resolved→03Open [19:45:20] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for BTracy-WMF - https://phabricator.wikimedia.org/T405366#11242778 (10Dzahn) 05Open→03Resolved ah:) cool. I did not mean to open it again. that was just because I had the tab alrea... [20:15:58] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242860 (10mforns) [20:19:09] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242864 (10mforns) [20:43:57] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11242933 (10mforns) [20:51:56] 06Data-Engineering: MediawikiPageContentChangeEnrichAvailability fired - https://phabricator.wikimedia.org/T406389 (10dr0ptp4kt) 03NEW [21:22:39] 06Data-Engineering: Clarify what to do with the 'Wp-zero-data-bot' account - https://phabricator.wikimedia.org/T406298#11243080 (10hashar) Note I have disabled the account in Gerrit. [21:55:54] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11243140 (10Ahoelzl) For the record, all backfills should be resumed. [22:39:36] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11243212 (10mforns)