[00:29:56] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work: Set up a working, usable dbt installation on stat boxes - https://phabricator.wikimedia.org/T406634#11445981 (10Ahoelzl) 05Open→03Resolved a:03Ahoelzl [00:29:58] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Add new data-steward email to Human-Bot Alert email. - https://phabricator.wikimedia.org/T411876#11445984 (10Ahoelzl) 05Open→03Resolved [00:29:58] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Human vs Bot Alerting Email Upgrade - https://phabricator.wikimedia.org/T411378#11445985 (10Ahoelzl) 05Open→03Resolved [00:30:01] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): change metric name for prometheus slo - https://phabricator.wikimedia.org/T411973#11445986 (10Ahoelzl) 05Open→03Resolved [00:30:03] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Implement Mediawiki Content History SLO monitoring and alerting - https://phabricator.wikimedia.org/T410579#11445988 (10Ahoelzl) 05Open→03Resolved [00:30:04] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 SPUR bot detection - Productionize SPUR datasets import - https://phabricator.wikimedia.org/T410285#11445989 (10Ahoelzl) 05Open→03Resolved [00:30:06] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Implement a new pipeline and table with reconciled historical revision data - https://phabricator.wikimedia.org/T410688#11445990 (10Ahoelzl) 05Open→03Resolved [00:30:10] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 Prepare dataset of IPs crossing Spur, legacy bot detection and HAProxy signals - https://phabricator.wikimedia.org/T408210#11445992 (10Ahoelzl) 05In progress→03Resolved [00:30:14] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Clean up airflow-dags gitlab-ci.yaml CI/CD pipelines - https://phabricator.wikimedia.org/T400283#11445995 (10Ahoelzl) 05Open→03Resolved [00:30:18] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Design and populate temporary table for JA3N analysis - https://phabricator.wikimedia.org/T408405#11445998 (10Ahoelzl) 05Open→03Resolved [00:30:23] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform, 13Patch-For-Review: Add CI step to event schema repositories to test to fail if a schema is deleted - https://phabricator.wikimedia.org/T377023#11445997 (10Ahoelzl) 05Open→03Resolved [00:30:26] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Engineering-Roadmap, 07Epic, 07OKR-Work, 13Patch-For-Review: Productionize JA3N-UA table to improve bot detection - https://phabricator.wikimedia.org/T409584#11445994 (10Ahoelzl) 05Open→03Resolved [00:30:31] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Reduce the number of refine job calls to the hive metastore - https://phabricator.wikimedia.org/T410378#11446000 (10Ahoelzl) 05Open→03Resolved [05:11:14] !log Test Kitchen edge-unique experiments (poll 1772) - adds: fy25-26-we-1-1-19-mobile-section-dead-end; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [05:11:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:08:27] (03CR) 10Joal: [V:03+2 C:03+2] Add compute_mediawiki_revision_history_reduced HQL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1214023 (https://phabricator.wikimedia.org/T411484) (owner: 10Joal) [09:08:34] (03PS4) 10Joal: Add compute_mediawiki_revision_history_reduced HQL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1214023 (https://phabricator.wikimedia.org/T411484) [09:08:44] (03CR) 10Joal: [V:03+2 C:03+2] Add compute_mediawiki_revision_history_reduced HQL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1214023 (https://phabricator.wikimedia.org/T411484) (owner: 10Joal) [09:24:44] !log Deploy refinery with Scap, and onto HDFS [09:24:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:18:31] 06Data-Engineering (Q3 2025 January 1st - March 31th), 06Product-Analytics, 10Event-Platform: Enable Event Platform streams to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#11446494 (10A_smart_kitten) @Ottomata @Sgs Apologies for the pings on this task from earlier thi... [10:41:41] 06Data-Engineering, 10DPE-Mediawiki-Content, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): When wikis cannot be exported due to SiteInfo, don't fail them - https://phabricator.wikimedia.org/T408819#11446549 (10brouberol) I've gotten so far as running the following query: ` from airflow.decorators import tas... [10:42:12] 06Data-Engineering, 10DPE-Mediawiki-Content, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): When wikis cannot be exported due to SiteInfo, don't fail them - https://phabricator.wikimedia.org/T408819#11446560 (10brouberol) a:05brouberol→03None [11:06:18] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Analyze and optimize Airflow Postgres backend performance - https://phabricator.wikimedia.org/T411990#11446639 (10Antoine_Quhen) Patch adding pg_stat to analytics_test https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1217138 [11:22:31] !log Test Kitchen edge-unique experiments (poll 2878) - adds: sticky-headers; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:22:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:47:15] 06Data-Engineering, 10DPE-Mediawiki-Content, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): When wikis cannot be exported due to SiteInfo, don't fail them - https://phabricator.wikimedia.org/T408819#11447589 (10xcollazo) Thanks for the effort @brouberol! Your error means that Hive really wants to force you... [14:49:26] 06Data-Engineering, 10DPE-Mediawiki-Content, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): When wikis cannot be exported due to SiteInfo, don't fail them - https://phabricator.wikimedia.org/T408819#11447612 (10brouberol) > Could we set a connection to a Spark Thrift Server? I don't think we have a Spark Thr... [15:00:03] 06Data-Engineering, 10DPE-Mediawiki-Content, 06Data-Platform-SRE (2025.11.07 - 2025.11.28): When wikis cannot be exported due to SiteInfo, don't fail them - https://phabricator.wikimedia.org/T408819#11447649 (10xcollazo) Yes, Presto is even better! [15:21:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10Event-Platform: mediawiki.page_change.v1 event stream - Investigate mistmatched meta.dt and dt (and rev_dt) fields - https://phabricator.wikimedia.org/T409105#11447766 (10JMonton-WMF) a:03JMonton-WMF [17:52:34] (03CR) 10Mforns: "LGMT! Left 2 minor comments, but ready to merge on my side!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1216874 (https://phabricator.wikimedia.org/T405039) (owner: 10Aleksandar Mastilovic) [20:44:38] (03PS2) 10Aleksandar Mastilovic: Add the monthly pageviews_per_editor HQL script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1216874 (https://phabricator.wikimedia.org/T405039) [20:44:49] (03CR) 10Aleksandar Mastilovic: "Ack" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1216874 (https://phabricator.wikimedia.org/T405039) (owner: 10Aleksandar Mastilovic) [20:46:05] (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1216874 (https://phabricator.wikimedia.org/T405039) (owner: 10Aleksandar Mastilovic) [21:05:21] (03CR) 10Aleksandar Mastilovic: [V:03+2] Add the monthly pageviews_per_editor HQL script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1216874 (https://phabricator.wikimedia.org/T405039) (owner: 10Aleksandar Mastilovic) [22:43:20] (03PS1) 10Aleksandar Mastilovic: Make pageviews per editor HQL scripts able to do idempotent runs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1217302 (https://phabricator.wikimedia.org/T405039) [22:44:45] (03PS2) 10Aleksandar Mastilovic: Make pageviews per editor HQL scripts able to do idempotent runs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1217302 (https://phabricator.wikimedia.org/T405039) [22:47:13] (03PS3) 10Aleksandar Mastilovic: Make pageviews per editor HQL scripts able to do idempotent runs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1217302 (https://phabricator.wikimedia.org/T405039) [22:50:16] (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1217302 (https://phabricator.wikimedia.org/T405039) (owner: 10Aleksandar Mastilovic) [22:50:20] (03CR) 10Aleksandar Mastilovic: [V:03+2] Make pageviews per editor HQL scripts able to do idempotent runs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1217302 (https://phabricator.wikimedia.org/T405039) (owner: 10Aleksandar Mastilovic)