[07:46:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:06:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:58:39] 06Data-Engineering-Radar, 10Research-engineering, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Add analytics-research user to stat boxes - https://phabricator.wikimedia.org/T403207#11245251 (10Stevemunene) 05Open→03Resolved [09:01:56] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Mitigate consequences of Gobblin hiccups generating late events and alerts - https://phabricator.wikimedia.org/T402324#11245279 (10JAllemandou) Thank you both for your answers :) I'll move forward with this. [13:08:47] (03CR) 10Ottomata: "Hi, thanks for the ping! Somehow I missed your earlier comment." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [13:12:16] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 10Event-Platform: Flink images should be build on top of openjdk - https://phabricator.wikimedia.org/T400296#11245979 (10Ottomata) 05Open→03Declined Being bold and declining. [13:20:26] (03PS2) 10Gkyziridis: expand_event_sanitized_analytics_allowlist: Add revert_risk prediction results to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) [13:20:36] (03CR) 10Gkyziridis: "Perfect, thank you so much." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [13:21:12] (03Abandoned) 10Gkyziridis: expand_event_sanitized_analytics_allowlist: Add outlink_topic prediction results to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192909 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [13:21:48] (03PS3) 10Gkyziridis: expand_event_sanitized_analytics_allowlist: Add revert_risk prediction results to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) [13:24:46] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Airflow jobs to do monthly XML dumps - https://phabricator.wikimedia.org/T384381#11246032 (10kevinbazira) Hi @xcollazo whenever you get a minute, please have a look at this [[ https://gitlab.wikimedia.org/... [13:30:45] (03CR) 10Aqu: [C:04-1] "Hi @gkyziridis@wikimedia.org and Thanks for the patch." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [13:32:19] (03CR) 10Aqu: [C:03+2] expand_event_sanitized_analytics_allowlist: Add revert_risk prediction results to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [13:37:36] (03CR) 10Gkyziridis: "Done" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [13:37:45] (03CR) 10Gkyziridis: [C:03+1] expand_event_sanitized_analytics_allowlist: Add revert_risk prediction results to allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192900 (https://phabricator.wikimedia.org/T405358) (owner: 10Gkyziridis) [14:42:10] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Airflow jobs to do monthly XML dumps - https://phabricator.wikimedia.org/T384381#11246378 (10kevinbazira) >>! In T384381#11246032, @kevinbazira wrote: > Hi @xcollazo whenever you get a minute, please have... [15:10:28] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: MediaWiki Content History alerts too much for minor reconcile issues - https://phabricator.wikimedia.org/T395139#11246526 (10Antoine_Quhen) [15:21:46] 06Data-Engineering, 06Data-Engineering-Radar, 06Traffic, 13Patch-For-Review: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11246578 (10CDanis) a:05CDanis→03JAllemandou joal please deploy [[ https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1192577 | re... [15:37:29] FYI, the "Designed by Aislinn Grigas" link at the bottom of stats.wikimedia.org goes to an inactive squarespace account [15:37:42] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it - https://phabricator.wikimedia.org/T388793#11246632 (10Antoine_Quhen) This is what I did to run the table copy after copying the t... [15:38:14] oh, there's a specific project in phab. I'll file a task [15:39:59] 10Data-Engineering-Wikistats: "Designed by Aislinn Grigas" at the bottom of stats.wikimedia.org leads to an inactive Squarespace account - https://phabricator.wikimedia.org/T406502 (10BCornwall) 03NEW [15:50:18] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11246721 (10isarantopoulos) >Update: Growth te... [15:52:59] (03PS1) 10Snwachukwu: Add th.wikimedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193888 [15:55:05] (03CR) 10Snwachukwu: [C:03+1] Add th.wikimedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193888 (owner: 10Snwachukwu) [15:57:47] (03CR) 10Dr0ptp4kt: [C:03+1] Add th.wikimedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193888 (owner: 10Snwachukwu) [16:22:11] (03PS1) 10Ottomata: hql/centralauth DDL - Fix location path for tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193897 (https://phabricator.wikimedia.org/T389666) [16:31:57] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11246935 (10Ottomata) {F66734036} WIP OpenApi Spec. See it here... [16:32:36] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: Global Editor Metrics - HTTP API endpoints - https://phabricator.wikimedia.org/T405041#11246936 (10Ottomata) [16:33:12] (03CR) 10Joal: [C:03+2] "LGTM! thanks for spotting" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193897 (https://phabricator.wikimedia.org/T389666) (owner: 10Ottomata) [16:34:05] (03PS2) 10Ottomata: sqoop centralauth - Fix location path for tables and name of sqoop tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193897 (https://phabricator.wikimedia.org/T389666) [16:34:19] (03CR) 10Snwachukwu: [C:03+2] Add th.wikimedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193888 (owner: 10Snwachukwu) [16:34:25] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Add th.wikimedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193888 (owner: 10Snwachukwu) [16:34:43] (03CR) 10Joal: [V:03+2 C:03+2] "Merging and applying manually" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1192577 (https://phabricator.wikimedia.org/T405783) (owner: 10CDanis) [16:39:46] (03PS3) 10Ottomata: sqoop centralauth - Fix location path for tables and name of sqoop tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193897 (https://phabricator.wikimedia.org/T389666) [16:40:53] (03CR) 10Ottomata: [C:03+2] sqoop centralauth - Fix location path for tables and name of sqoop tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193897 (https://phabricator.wikimedia.org/T389666) (owner: 10Ottomata) [16:41:51] (03CR) 10Ottomata: [V:03+2 C:03+2] sqoop centralauth - Fix location path for tables and name of sqoop tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193897 (https://phabricator.wikimedia.org/T389666) (owner: 10Ottomata) [16:42:49] (03CR) 10Joal: "Two changes needed" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1193897 (https://phabricator.wikimedia.org/T389666) (owner: 10Ottomata) [16:44:12] !log deploying analytics/refinery to an-launcher1002 to pick up change for T389666 [16:44:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:44:24] T389666: NEW/CHANGE FEATURE REQUEST: make available the centralauth.globaluser table in Data Lake - https://phabricator.wikimedia.org/T389666 [16:47:54] 06Data-Engineering, 06Data-Engineering-Radar, 06Traffic, 13Patch-For-Review: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11246994 (10JAllemandou) Ingestion job restarted with the new sttings. The new field should start to populate soon. [16:51:26] 06Data-Engineering: Commons Impact Metrics has no data for September snapshot - https://phabricator.wikimedia.org/T406509 (10mforns) 03NEW [16:52:59] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Experimentation Lab, 13Patch-For-Review: NEW/CHANGE FEATURE REQUEST: make available the centralauth.globaluser table in Data Lake - https://phabricator.wikimedia.org/T389666#11247012 (10Ottomata) Created tables. ` sudo -u analytics kerberos-run-co... [16:53:28] 06Data-Engineering, 10Data-Engineering-Wikistats: "Designed by Aislinn Grigas" at the bottom of stats.wikimedia.org leads to an inactive Squarespace account - https://phabricator.wikimedia.org/T406502#11247017 (10Hemmanttt) a:03Hemmanttt [16:55:58] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Commons Impact Metrics has no data for September snapshot - https://phabricator.wikimedia.org/T406509#11247048 (10mforns) [18:13:43] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11247231 (10Ottomata) In a [[ https://www.google.com/calendar/event?eid=M25nMnM2bTdvbzloMmk4dThhNDV... [18:14:38] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Data Pipeline - https://phabricator.wikimedia.org/T405039#11247237 (10Ottomata) [18:23:37] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Add user_central_id to mediawiki_content_history_v1 (and mediawiki_content_current_v1) - https://phabricator.wikimedia.org/T406515 (10Ottomata) 03NEW [18:25:37] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Add user_central_id to mediawiki_content_history_v1 (and mediawiki_content_current_v1) - https://phabricator.wikimedia.org/T406515#11247289 (10Ottomata) @xcollazo, @Antoine_Quhen, in a [[ https://docs.google... [18:27:46] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Data Pipeline - https://phabricator.wikimedia.org/T405039#11247302 (10Ottomata) Relevant Decisions from a [[ https://docs.google.com/document/d/19ErSFYSUCeRl8hcMZDfzxoRYwUxgIjL-QuOksA... [18:29:45] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Page-derived-data, 07OKR-Work: Add user_central_id to mediawiki_content_history_v1 (and mediawiki_content_current_v1) - https://phabricator.wikimedia.org/T406515#11247309 (10Ottomata) A summary of the decision from the meeting is at {T405... [19:28:19] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Wikimedia Enterprise, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11247554 (10Ottomata) [19:28:40] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Wikimedia Enterprise, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11247557 (10Ottomata) cc @Htriedman I think there m... [19:55:09] 06Data-Engineering, 06Data-Engineering-Radar, 06Traffic, 13Patch-For-Review: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11247628 (10CDanis) 05Open→03Resolved [20:53:12] 06Data-Engineering: mediawiki_history_reduced.check_mediawiki_history_reduced error ratio hit - https://phabricator.wikimedia.org/T406525 (10dr0ptp4kt) 03NEW [20:55:40] 06Data-Engineering: mediawiki_history_reduced.check_mediawiki_history_reduced error ratio hit - https://phabricator.wikimedia.org/T406525#11247813 (10dr0ptp4kt) [21:02:34] 06Data-Engineering: mediawiki_history_reduced.check_mediawiki_history_reduced error ratio hit - https://phabricator.wikimedia.org/T406525#11247825 (10dr0ptp4kt) [21:15:30] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Serve mobile and desktop variants through the same URL (unified mobile routing) - https://phabricator.wikimedia.org/T214998#11247834 (10Krinkle) [21:17:01] 06Data-Engineering: GobblinLastSuccessfulRunTooLongAgo alerts - https://phabricator.wikimedia.org/T406526#11247856 (10dr0ptp4kt) [22:01:25] 06Data-Engineering: NEW BUG REPORT : Referer Name field is Null on Druid pageviews_daily - https://phabricator.wikimedia.org/T406530 (10Mayakp.wiki) 03NEW [22:05:46] 06Data-Engineering: NEW BUG REPORT : Referer Name field is Null on Druid pageviews_daily - https://phabricator.wikimedia.org/T406530#11247965 (10Mayakp.wiki) hmm, apparently this was done as a part of T334104#8759879, but I wonder if we encountered a bug later and the data stopped loading from hive to druid? [22:08:20] 06Data-Engineering, 06Movement-Insights: NEW BUG REPORT : Referer Name field is Null on Druid pageviews_daily - https://phabricator.wikimedia.org/T406530#11247967 (10Mayakp.wiki) [22:08:25] 06Data-Engineering, 06Movement-Insights: NEW BUG REPORT : Referer Name field is Null on Druid pageviews_daily - https://phabricator.wikimedia.org/T406530#11247968 (10Mayakp.wiki) p:05Triage→03High [22:24:46] 06Data-Engineering, 10Data-Platform, 06Movement-Insights: NEWFEATURE REQUEST: - https://phabricator.wikimedia.org/T406531 (10Mayakp.wiki) 03NEW [22:31:06] 06Data-Engineering, 10Data-Platform, 06Movement-Insights: NEWFEATURE REQUEST: Update referers - https://phabricator.wikimedia.org/T406531#11248002 (10Pppery) [22:35:37] 06Data-Engineering, 10Data-Platform, 06Movement-Insights: NEWFEATURE REQUEST: Update referers - https://phabricator.wikimedia.org/T406531#11248030 (10Mayakp.wiki) p:05Triage→03High [22:35:59] 06Data-Engineering, 10Data-Platform, 06Movement-Insights: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11248031 (10Mayakp.wiki) [22:46:41] 06Data-Engineering: GobblinLastSuccessfulRunTooLongAgo alerts - https://phabricator.wikimedia.org/T406526#11248044 (10dr0ptp4kt)