[02:31:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07Epic, 07OKR-Work: SDS 1.3.2 [EPIC] Automated alerting for changes in automated traffic behavior - https://phabricator.wikimedia.org/T407235#11321916 (10Ahoelzl) [02:47:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07Epic, 07OKR-Work: SDS 1.3.6 Improved bot detection using Spur data set - https://phabricator.wikimedia.org/T408656 (10Ahoelzl) 03NEW [02:48:47] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 SPUR bot detection analysis - https://phabricator.wikimedia.org/T407103#11321928 (10Ahoelzl) [02:48:50] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07Epic, 07OKR-Work: SDS 1.3.6 Improved bot detection using Spur data set - https://phabricator.wikimedia.org/T408656#11321929 (10Ahoelzl) [02:57:13] 06Data-Engineering, 13Patch-For-Review: Enable Spark data lineage for all Airflow instances - https://phabricator.wikimedia.org/T386862#11321930 (10Ahoelzl) [02:59:30] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07Epic, 07OKR-Work: Technical work for SDS1.3.7 Incorporate Edge Signal - https://phabricator.wikimedia.org/T407893#11321933 (10Ahoelzl) [05:43:04] 06Data-Engineering, 10ChangeProp, 10Citoid, 06Editing-team, and 19 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11322046 (10Physikerwelt) [07:19:49] (03PS5) 10Aqu: Maintain Spark schema after Hive DDL operations [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199473 (https://phabricator.wikimedia.org/T307040) [08:28:10] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11322206 (10dcausse) >>! In T407514#11320956, @TJones wrote: > I often... [08:41:16] (03PS6) 10Aqu: Maintain Spark schema after Hive DDL operations [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199473 (https://phabricator.wikimedia.org/T307040) [08:42:43] (03CR) 10Aqu: Maintain Spark schema after Hive DDL operations (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199473 (https://phabricator.wikimedia.org/T307040) (owner: 10Aqu) [10:11:03] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11322473 (10dcausse) >>! In T407514#11322446, @Gehel wrote: >>>! In T40... [12:14:55] !log roll restart druid worker hosts for T408189 [12:14:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:14:58] T408189: Increase the size of the Druid broker cache size from 2GB to 4GB - https://phabricator.wikimedia.org/T408189 [12:23:43] 06Data-Engineering, 10ChangeProp, 10EventStreams, 10iPoid-Service, and 17 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11322892 (10Mvolz) [13:11:59] 06Data-Engineering, 06Data-Platform-SRE, 07Essential-Work, 13Patch-For-Review: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11322978 (10Gehel) [13:12:00] 06Data-Engineering, 06Data-Platform-SRE, 07Essential-Work, 13Patch-For-Review: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11322979 (10Gehel) 05Open→03Resolved [13:12:02] (03CR) 10Ottomata: Maintain Spark schema after Hive DDL operations (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199473 (https://phabricator.wikimedia.org/T307040) (owner: 10Aqu) [13:12:35] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work, 13Patch-For-Review: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11322985 (10Gehel) [13:29:06] 06Data-Engineering, 06SRE: stat1011: cannot create directory ‘/srv/published/datasets/one-off’: Permission denied - https://phabricator.wikimedia.org/T408641#11323056 (10Ottomata) ` 13:26:24 [@stat1011:/home/otto] $ ls -la /srv/published/ total 28 drwxrwxr-x 6 root wikidev 4096 Oct 31 2024 . dr... [13:50:25] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE, 06Movement-Insights, 07Epic: Create example models using Iceberg - https://phabricator.wikimedia.org/T408687 (10JMonton-WMF) 03NEW [13:54:49] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: SDS 1.3.2 Conduct Analysis on Alerting for changes in automated traffic distribution - https://phabricator.wikimedia.org/T406882#11323124 (10Snwachukwu) Yes, for clarity, Here is the approach decided on to define our thresholds: - Use Fix... [14:05:13] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10AbuseFilter, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997#11323199 (10Marostegui) [14:13:41] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Increase the size of the Druid broker cache size from 2GB to 4GB - https://phabricator.wikimedia.org/T408189#11323251 (10Stevemunene) 05Open→03Resolved Ran puppet on the hosts then restarted the druid daemons ` stev... [14:34:02] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement the data layout, UI, and documentation for the XML file export - https://phabricator.wikimedia.org/T401022#11323331 (10xcollazo) Ran the following to remove older test runs: ` hdfs dfs -rm -r -skipTrash /wmf/data/exports... [14:34:38] 06Data-Engineering: Requesting Kerberos access for slyngshede - https://phabricator.wikimedia.org/T408696 (10SLyngshede-WMF) 03NEW [15:05:33] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE, 06Movement-Insights, 07Epic: Create example models using Iceberg - https://phabricator.wikimedia.org/T408687#11323502 (10JMonton-WMF) Created a MR with a couple of examples of Iceberg tables. https://gitlab.wikimedia.org/rep... [15:10:17] 06Data-Engineering, 06SRE: stat1011: cannot create directory ‘/srv/published/datasets/one-off’: Permission denied - https://phabricator.wikimedia.org/T408641#11323570 (10Addshore) 05Open→03Resolved a:03Addshore Success! ty [15:15:37] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11323602 (10fnegri) [15:15:42] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE, 06Movement-Insights, 07Epic: Create example dbt models using Iceberg - https://phabricator.wikimedia.org/T408687#11323604 (10Ottomata) [15:24:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11323655 (10Gehel) `false` does not seem to... [15:25:27] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11323657 (10Gehel) Change is merged, we need to publish a new release o... [16:03:09] 06Data-Engineering, 06Data-Engineering-Radar: Requesting Kerberos access for slyngshede - https://phabricator.wikimedia.org/T408696#11323874 (10Ottomata) [16:03:20] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Requesting Kerberos access for Jmoore111 - https://phabricator.wikimedia.org/T408165#11323876 (10Ottomata) [16:11:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Movement-Insights, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Epic: Create example dbt models using Iceberg - https://phabricator.wikimedia.org/T408687#11323927 (10Gehel) [17:40:39] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Update MediaWiki Content History SLO draft for SRE review - https://phabricator.wikimedia.org/T401892#11324457 (10APizzata-WMF) a:05xcollazo→03APizzata-WMF [17:51:13] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11324507 (10achou) Trying to address the blo... [18:06:24] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11324575 (10Michael) @Eevans and I just had... [18:09:01] (03PS1) 10Snwachukwu: Update change log for v0.3.6 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199835 [18:11:39] !log deploying refinery source as part of deployment train. [18:11:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:17:24] (03PS17) 10Ottomata: Add HQL for edit_per_editor_per_page_daily and pageview_per_editor_per_page_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [18:23:37] (03CR) 10Snwachukwu: [C:03+2] Update change log for v0.3.6 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199835 (owner: 10Snwachukwu) [18:24:02] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Update change log for v0.3.6 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199835 (owner: 10Snwachukwu) [18:25:14] Starting build #51 for job analytics-refinery-maven-release [18:49:02] Project analytics-refinery-maven-release build #51: 09SUCCESS in 23 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/51/ [18:55:34] Starting build #38 for job analytics-refinery-update-jars [18:56:13] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.3.6 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1199840 [18:56:14] Project analytics-refinery-update-jars build #38: 09SUCCESS in 39 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/38/ [19:14:05] (03CR) 10Snwachukwu: [C:03+2] Add refinery-source jars for v0.3.6 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1199840 (owner: 10Maven-release-user) [19:14:09] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Add refinery-source jars for v0.3.6 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1199840 (owner: 10Maven-release-user) [19:16:26] !log Deployed refinery-source [19:16:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:26:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11324847 (10Eevans) [19:37:50] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement the data layout, UI, and documentation for the XML file export - https://phabricator.wikimedia.org/T401022#11324943 (10xcollazo) Rerunning `mw_content_xml_export_current_mid_month` for `2025-10-15` to double check the ch... [19:42:28] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 07Essential-Work, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11324961 (10Mayakp.wiki) @JAllemandou , do we hav... [20:19:12] (03PS3) 10Snwachukwu: Add Data quality check for Pageview Human-Bot ratio anomaly [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199485 (https://phabricator.wikimedia.org/T407239) [20:20:07] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11325074 (10Eevans) >>! In T401021#11324507,... [20:20:15] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Requesting Kerberos access for Jmoore111 - https://phabricator.wikimedia.org/T408165#11325080 (10RKemper) Configured a kerberos principal (hopefully I was supposed to do that in this ticket and no... [20:25:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11325097 (10Eevans) [20:25:49] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11325119 (10Eevans) [20:30:14] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Requesting Kerberos access for Jmoore111 - https://phabricator.wikimedia.org/T408165#11325139 (10RKemper) Oops, needed to have made it for `jmoore111`. I deleted the old principal and recreated wi... [20:38:03] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Requesting Kerberos access for Jmoore111 - https://phabricator.wikimedia.org/T408165#11325163 (10RKemper) 05Open→03Resolved Okay, we verified that the kerberos principal is set up and Just... [20:50:57] 06Data-Engineering, 06Data-Engineering-Radar, 10SRE-Access-Requests, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Requesting Kerberos access for Jmoore111 - https://phabricator.wikimedia.org/T408165#11325192 (10Dzahn) [21:36:35] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Movement-Insights: Improve referrer tracking/classification using `utm_source` URL parameter - https://phabricator.wikimedia.org/T408185#11325348 (10Ahoelzl) [21:37:21] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Movement-Insights: Improve referrer tracking/classification using `utm_source` URL parameter - https://phabricator.wikimedia.org/T408185#11325356 (10Ahoelzl) p:05Triage→03Medium a:03JAllemandou [21:37:40] 06Data-Engineering, 07Epic, 07OKR-Work: SDS 1.3.6 Improved bot detection using Spur data set - https://phabricator.wikimedia.org/T408656#11325368 (10Ahoelzl) [21:37:53] 06Data-Engineering, 07Epic, 07OKR-Work: Technical work for SDS1.3.7 Incorporate Edge Signal - https://phabricator.wikimedia.org/T407893#11325381 (10Ahoelzl) [21:38:30] 10Data-Engineering-Roadmap, 07Epic, 07OKR-Work: SDS 1.3.6 Improved bot detection using Spur data set - https://phabricator.wikimedia.org/T408656#11325386 (10Ahoelzl) [21:38:35] 10Data-Engineering-Roadmap, 07Epic, 07OKR-Work: Technical work for SDS1.3.7 Incorporate Edge Signal - https://phabricator.wikimedia.org/T407893#11325388 (10Ahoelzl) [21:39:30] 10Data-Engineering-Roadmap, 07Epic: [Spike] Evaluate AI tooling for data engineering - https://phabricator.wikimedia.org/T391422#11325396 (10Ahoelzl) 05Open→03Resolved a:03Ahoelzl [21:39:33] 10Data-Engineering-Roadmap, 10Data Pipelines, 07Epic: Refine jobs should be scheduled by Airflow - https://phabricator.wikimedia.org/T307505#11325398 (10Ahoelzl) 05In progress→03Resolved a:03Ahoelzl [21:39:35] 10Data-Engineering-Roadmap, 06Product Safety and Integrity, 06Product-Analytics, 10Temporary accounts, 07Epic: [Epic] Update schemas and instrumentation code for temporary accounts - https://phabricator.wikimedia.org/T374942#11325394 (10Ahoelzl) 05Open→03Resolved a:03Ahoelzl [21:44:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 07Essential-Work, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11325444 (10Ahoelzl) @Mayakp.wiki The extended we... [23:42:37] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06SRE: Move Druid realtime configuration out of Refinery into standalone repo on GitLab - https://phabricator.wikimedia.org/T407994#11325805 (10amastilovic) > Do we want only Druid realtime configs its own repo? Perhaps we want the batch ones in the...