[00:26:44] 06Data-Engineering, 06Data-Platform-SRE, 06Traffic, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11170582 (10Ottomata) I mean, we want the Kafka logging mirror maker piece anyway for T304373, but I think w... [06:54:06] 06Data-Engineering, 06SRE, 06Traffic: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11170835 (10JAllemandou) [08:14:58] 06Data-Engineering, 06Traffic: Reduce noise from duplicate sequence-gap alerts on HaProxy-webrequests - https://phabricator.wikimedia.org/T401383#11171106 (10Fabfur) [08:17:39] 06Data-Engineering, 10Data Pipelines: Provide an easy way for MediaWiki to fetch aggregate data from the data lake - https://phabricator.wikimedia.org/T341649#11171120 (10Addshore) [09:29:04] !log add druid1012 to druid_public cluster T397441 [09:29:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:29:11] T397441: Bring druid101[2-3] online to replace druid100[7-8] - https://phabricator.wikimedia.org/T397441 [09:38:32] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11171403 (10Ladsgroup) [09:56:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:01:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:06:17] FIRING: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:08:38] !log starting deployment of airflow-dags for https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1623 [10:11:17] RESOLVED: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:29:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:59:48] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:28:44] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11171934 (10Ladsgroup) [11:49:16] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Refine to Hive with Airflow – Update Refine Documentation on Wikitech - https://phabricator.wikimedia.org/T392697#11171998 (10Antoine_Quhen) 05Open→03Resolved Closing. [11:53:24] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11172013 (10Ladsgroup) [12:09:22] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking), 10Wikipedia-iOS-App-Backlog (iOS Release FY2025-26): [QA Task] Verify iOS compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404275#11172095 (10Seddon) [12:10:45] 06Data-Engineering, 10Data Pipelines, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work, 13Patch-For-Review: AirFlow DAGs: unable to publish deb via CI - https://phabricator.wikimedia.org/T403016#11172101 (10Gehel) [12:10:50] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11172104 (10Gehel) [12:11:43] 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work: Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11172121 (10Gehel) [12:11:57] 06Data-Engineering-Radar, 10Research-engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work: Add analytics-research user to stat boxes - https://phabricator.wikimedia.org/T403207#11172123 (10Gehel) [12:12:03] 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11172126 (10Gehel) [12:12:15] 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work: Request for dedicated Airflow instance for WME - https://phabricator.wikimedia.org/T396672#11172130 (10Gehel) [12:12:42] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 7 others: Title of suppressed recentchanges entries can be viewed through the wiki replicas - https://phabricator.wikimedia.org/T402283#11172124 (10Gehel) [12:13:08] 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work: Fix the hard dependency between the Airflow scheduler and the DataHub GMS service - https://phabricator.wikimedia.org/T395106#11172135 (10Gehel) [12:13:43] 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, and 3 others: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#11172141 (10Gehel) [12:14:05] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), and 2 others: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#11172147 (10Gehel) [12:17:19] 06Data-Engineering, 10Technical-blog-posts, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work: Write a blog post about the recent Airflow migration to Kubernetes - https://phabricator.wikimedia.org/T393603#11172202 (10Gehel) [12:17:29] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11172204 (10brouberol) @JAllemandou @dcausse and myself looked... [12:17:43] 06Data-Engineering, 06Data-Engineering-Radar, 06cloud-services-team, 06Data-Persistence, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11172207 (10Gehel) [12:20:35] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking), 10Wikipedia-iOS-App-Backlog (iOS Release FY2025-26): [QA Task] Verify iOS compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404275#11172263 (10Seddon) [12:20:43] 06Data-Engineering, 06Java-Scala-Standardization, 10Discovery-Search (2025.09.05 - 2025.09.26), 07Essential-Work: Create Gitlab CI templates for JVM packages - https://phabricator.wikimedia.org/T386406#11172290 (10Gehel) [12:21:17] 06Data-Engineering, 10CirrusSearch, 10DPE-Mediawiki-Content, 10Discovery-Search (2025.09.05 - 2025.09.26), and 2 others: Source the CirrusSearch index dumps from hadoop instead of a MW maintenance script - https://phabricator.wikimedia.org/T366248#11172293 (10Gehel) [12:21:37] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 10Discovery-Search (2025.09.05 - 2025.09.26), and 2 others: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies ... - https://phabricator.wikimedia.org/T367405#11172298 [12:21:43] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking), 10Wikipedia-iOS-App-Backlog (iOS Release FY2025-26): [QA Task] Verify iOS compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404275#11172291 (10Seddon) [12:23:42] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking), 10Wikipedia-iOS-App-Backlog (iOS Release FY2025-26): [QA Task] Verify iOS compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404275#11172323 (10Seddon) [12:32:24] 06Data-Engineering, 06Discovery-Search, 07Epic, 07Essential-Work: [EPIC] Upgrade flink jobs to java 17 - https://phabricator.wikimedia.org/T404340 (10dcausse) 03NEW [12:32:54] 06Data-Engineering, 06Discovery-Search, 07Epic, 07Essential-Work: [EPIC] Upgrade flink jobs to java 17 - https://phabricator.wikimedia.org/T404340#11172355 (10dcausse) [12:32:58] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11172356 (10dcausse) [12:40:23] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): [QA Task] Verify Android compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404342 (10Seddon) 03NEW [12:40:24] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking), 10Wikipedia-iOS-App-Backlog (iOS Release FY2025-26): [QA Task] Verify iOS compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404275#11172391 (10Seddon) [12:55:44] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11172458 (10Ladsgroup) [13:05:21] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): [QA Task] Verify Android compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404342#11172513 (10402998) a:03402998 Verified Working https://www.mediawiki.org/wiki/Requests_for_comment/Mobil... [13:06:41] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11172517 (10Ladsgroup) [13:53:55] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10MediaWiki-Page-derived-data, and 2 others: Normalize categorylinks table - https://phabricator.wikimedia.org/T299951#11172732 (10hashar) [14:11:08] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10MediaWiki-Page-derived-data, and 2 others: Normalize categorylinks table - https://phabricator.wikimedia.org/T299951#11172856 (10Ladsgroup) [14:23:08] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Experimentation Lab, 10Event-Platform: EventGate: Log unparseable X-Experiment-Enrollments headers to an error stream - https://phabricator.wikimedia.org/T396359#11172993 (10dr0ptp4kt) 05Open→03Resolved I filed {T404352}. Thus, closing this... [15:08:20] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11173285 (10brouberol) ` * docker-registry.discovery.wmnet/fli... [15:08:42] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11173286 (10brouberol) I'll let this tickeyt open and mark it... [15:32:11] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): [QA Task] Verify Android compatability with removal of m. subdomain on test wiki - https://phabricator.wikimedia.org/T404342#11173394 (10Krinkle) a:05402998→03None [15:59:40] 06Data-Engineering: Ops Week: pageview_actor_hourly DAG is failing - https://phabricator.wikimedia.org/T404372 (10BTullis) 03NEW [15:59:53] 06Data-Engineering: Ops Week: pageview_actor_hourly DAG is failing - https://phabricator.wikimedia.org/T404372#11173544 (10BTullis) p:05Triage→03High [16:17:59] 06Data-Engineering, 06Experimentation Lab, 10MediaWiki-extensions-WikimediaEvents: Prepare mediawiki-client-error Logstash dashboards for mobile subdomain sunsetting - https://phabricator.wikimedia.org/T400852#11173641 (10Jdlrobson-WMF) The owner is listed as Data Products in https://github.com/wikimedia/med... [17:24:32] (03PS3) 10Joal: Add access_method to unique-devices per-domain [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1186087 (https://phabricator.wikimedia.org/T401666) [18:21:25] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): QA features on the new mobile URLs - https://phabricator.wikimedia.org/T403638#11174010 (10jsn.sherman) [19:30:43] 06Data-Engineering, 06SRE, 06Traffic: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11174218 (10CDanis) Thank you @JAllemandou ! This doesn't look too hard to implement in bloblang -- thankfully the definition of [[ htt... [22:01:16] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: Global Editor Metrics for YiR, Apps Activity Tab, and Growth Impact Module - https://phabricator.wikimedia.org/T403660#11174559 (10KStoller-WMF) >So, what's a good expirat...