[07:11:51] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11232083 (10JAllemandou) This is great, thank you for the writing Andrew :) I have two comments: *... [08:44:39] (03PS3) 10Aqu: WIP: Script to manually add a map-column to mediawiki_content_history_v1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1191982 (https://phabricator.wikimedia.org/T388793) [12:04:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:09:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:01:29] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Adapt mediawiki_history to the removal of mediawiki revision.rev_sha1 - https://phabricator.wikimedia.org/T406000#11233074 (10JAllemandou) > shifting MW history down a level to think about slots instead of revisions. Then we wouldn't have to compute this... [13:19:41] 07Analytics-Data-Problem, 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Unique devices data has rows without any data - https://phabricator.wikimedia.org/T405430#11233142 (10JAllemandou) This is all done, no more rows with 0 `uniques_estimate` on those tables :) [13:32:33] Starting build #33 for job analytics-refinery-update-jars [13:33:24] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.3.3 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192888 [13:33:24] Project analytics-refinery-update-jars build #33: 09SUCCESS in 51 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/33/ [13:38:20] (03CR) 10Snwachukwu: [C:03+2] Add refinery-source jars for v0.3.3 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192888 (owner: 10Maven-release-user) [13:38:22] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Add refinery-source jars for v0.3.3 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192888 (owner: 10Maven-release-user) [13:44:49] !log Deployed refinery-source using jenkins(weekly deployment train) [13:44:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:12:11] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11233442 (10Ladsgroup) [14:15:10] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11233457 (10Ladsgroup) [14:22:36] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11233494 (10Ottomata) Thanks @JAllemandou, I think one of the biggest advantages of Option 2 is tha... [14:32:16] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11233527 (10Ladsgroup) [14:32:49] (03PS1) 10Gkyziridis: expand_event_sanitized_analytics_allowlist: Add revert_risk prediction results to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) [14:35:54] 06Data-Engineering, 06Release-Engineering-Team, 10Continuous-Integration-Config: Unable to publish to Maven - https://phabricator.wikimedia.org/T406072#11233541 (10brennen) 05In progress→03Resolved p:05Triage→03Medium Going ahead and resolving this, please reopen if anything breaks with those jobs. [14:51:02] (03CR) 10Ottomata: "Hi! thanks!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [14:57:05] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11233592 (10Milimetric) I vote Option 2. If we need to rollback a snapshot, wouldn't we simply: *... [15:02:36] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11233604 (10Ottomata) @JTannerWMF Prod... [15:03:52] (03CR) 10Gkyziridis: "Apologies, first time committing on this repo." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [15:17:24] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11233656 (10Ladsgroup) [15:17:30] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11233657 (10Ladsgroup) 05Open→03Resolved [15:17:50] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11233661 (10Ladsgroup) [15:18:50] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11233665 (10Ladsgroup) [15:19:24] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11233669 (10Ladsgroup) [15:20:58] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11233676 (10Ladsgroup) [15:23:51] (03PS1) 10Gkyziridis: expand_event_sanitized_analytics_allowlist: Add outlink_topic prediction results to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192909 (https://phabricator.wikimedia.org/T405358) [15:24:42] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11233686 (10Ladsgroup) [15:25:20] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11233691 (10Ladsgroup) [15:25:51] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11233693 (10Ladsgroup) [15:26:52] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11233701 (10Ladsgroup) 05In progress→03Resolved [16:12:15] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: [Spike] Comparitive game stats - https://phabricator.wikimedia.org/T405865#11233899 (10Ahoelzl) [16:12:49] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: [Spike] Comparitive game stats - https://phabricator.wikimedia.org/T405865#11233900 (10Ahoelzl) a:03Ahoelzl [16:13:08] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Provide an access to MaxMind GeoIP in DSE K8S pods - https://phabricator.wikimedia.org/T405509#11233906 (10Ottomata) [16:17:58] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Provide an access to MaxMind GeoIP in DSE K8S pods - https://phabricator.wikimedia.org/T405509#11233939 (10Ahoelzl) @BTullis we briefly discussed an alternative pathway for making the data accessi... [16:20:39] 06Data-Engineering, 10Event-Platform: Schema validation for EventStreamConfig - https://phabricator.wikimedia.org/T405516#11233949 (10Ottomata) p:05Triage→03Medium [16:23:04] 07Analytics-Data-Problem, 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Unique devices data uses non-standard domains for Wikidata, Wikifunctions, and MediaWiki.org - https://phabricator.wikimedia.org/T405533#11233954 (10Ottomata) a:03JAllemandou Grooming: Joseph, we assigned this to you, feel f... [16:23:07] 07Analytics-Data-Problem, 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Unique devices data uses non-standard domains for Wikidata, Wikifunctions, and MediaWiki.org - https://phabricator.wikimedia.org/T405533#11233957 (10Ottomata) [16:25:49] 06Data-Engineering-Radar, 06Data-Platform-SRE, 06Product-Analytics, 10superset.wikimedia.org: Improve Superset's error message for draft dashboards - https://phabricator.wikimedia.org/T405535#11233973 (10Ottomata) p:05Triage→03Low Grooming: Dear SRE team, if you have ideas on how to resolve, please do!... [16:27:50] 06Data-Engineering, 06Traffic, 13Patch-For-Review: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11233996 (10Ottomata) @JAllemandou to review patches druid ingestion patch? [16:28:22] 06Data-Engineering, 06Data-Engineering-Radar, 06Traffic, 13Patch-For-Review: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11234002 (10Ottomata) [16:33:01] 06Data-Engineering, 10Observability-Alerting, 10Event-Platform: EventgateProduceRateStop alert should be active datacenter aware - https://phabricator.wikimedia.org/T405952#11234015 (10Ottomata) a:03tchin Grooming: @tchin we are hoping you can find some time to look into this. This is an event platform rel... [16:33:08] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Observability-Alerting, 10Event-Platform: EventgateProduceRateStop alert should be active datacenter aware - https://phabricator.wikimedia.org/T405952#11234018 (10Ottomata) [16:36:02] 06Data-Engineering, 06Infrastructure-Foundations, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 13Patch-For-Review: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11234021 (10colewhite) >>! In T304373#11229932, @Ottomata wrote: > Can you a... [16:36:56] 06Data-Engineering, 10Wikidata, 10Wikidata-Query-Service, 10Event-Platform: WDQS sparql query event generator should not set meta.dt - https://phabricator.wikimedia.org/T405949#11234026 (10Ottomata) Grooming: we are going to put this in DE radar and hope that Search can fix this soonish. It isn't super urg... [16:45:14] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11234044 (10Dzahn) 05In progress→03Stalled stalled on manager approval. [17:19:36] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11234267 (10Krinkle) [17:57:32] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Adapt mediawiki_history to the removal of mediawiki revision.rev_sha1 - https://phabricator.wikimedia.org/T406000#11234528 (10Ottomata) Hi! Since we are making changes to MW history, could we do {T365648} at the same time? We need this for {T403660}, an... [18:22:06] 06Data-Engineering, 06Infrastructure-Foundations, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 13Patch-For-Review: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373#11234617 (10Ottomata) Sweet! Let's do it! Submitted a patch but I'll let ya... [18:27:33] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11234645 (10Ottomata) 2 things to check before we proceed: 1. Will druid handle these queries unde... [18:59:09] 06Data-Engineering, 10Wikidata, 10Wikidata-Query-Service, 10Event-Platform: WDQS sparql query event generator should not set meta.dt - https://phabricator.wikimedia.org/T405949#11234737 (10dcausse) >>! In T405949#11234025, @Ottomata wrote: > Grooming: we are going to put this in DE radar and hope that Sear... [19:40:28] 06Data-Engineering, 06Moderator-Tools-Team, 10PageTriage, 07Schema-change: refactor code to not use pagetriage_tags table. hard code it in PHP instead. - https://phabricator.wikimedia.org/T406177 (10Novem_Linguae) 03NEW [19:41:17] 06Data-Engineering, 06Moderator-Tools-Team, 10PageTriage, 07Schema-change: refactor code to not use pagetriage_tags table. hard code it in PHP instead. - https://phabricator.wikimedia.org/T406177#11234868 (10Novem_Linguae) [19:42:19] 06Data-Engineering, 06Moderator-Tools-Team, 10PageTriage, 07Schema-change: refactor code to not use pagetriage_tags table. hard code it in PHP instead. - https://phabricator.wikimedia.org/T406177#11234871 (10Novem_Linguae) [20:10:38] 06Data-Engineering, 10Wikidata, 10Wikidata-Query-Service, 10Event-Platform: WDQS sparql query event generator should not set meta.dt - https://phabricator.wikimedia.org/T405949#11234946 (10Ottomata) Nothing bad is happening on our end, except that we can't remove o11y's logstash filter for this logspam unt...