[00:04:39] 06Data-Engineering, 10CirrusSearch, 06Data-Platform-SRE, 10DPE-Mediawiki-Content, and 4 others: Source the CirrusSearch index dumps from hadoop instead of a MW maintenance script - https://phabricator.wikimedia.org/T366248#11316596 (10Ottomata) [01:16:42] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Fix issues with mediawiki_history_reduced table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1198607 (https://phabricator.wikimedia.org/T406263) (owner: 10Aleksandar Mastilovic) [07:21:48] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Increase the size of the Druid broker cache size from 2GB to 4GB - https://phabricator.wikimedia.org/T408189#11317425 (10Stevemunene) a:03Stevemunene [07:43:32] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06DBA, and 2 others: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587#11317465 (10Marostegui) 05Open→03Resolved All done [08:05:38] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11317489 (10Gehel) p:05Triage→03High [08:05:51] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11317491 (10Gehel) [08:21:59] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11317534 (10brouberol) @bking I'd say go for it 👍 [08:22:24] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11317537 (10Gehel) Checked on a random host, there are > 5G of tmp files older than 30 days: gehel@an-worker1161:~$ sud... [08:39:07] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11317632 (10brouberol) In the meantime I ran ` brouberol@cumin1003:~$ sudo cumin A:hadoop-worker 'find /tmp -name *.jar -mti... [08:51:27] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11317658 (10Gehel) a:03Gehel [10:22:20] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Product-Analytics, 10Event-Platform, 13Patch-For-Review: [Event Platform] Disable default collection of user agent for analytics streams - https://phabricator.wikimedia.org/T384964#11318053 (10JMonton-WMF) The Gerrit patch https://gerrit.wikime... [10:29:53] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11318094 (10dcausse) @TJones could you try to replicate this step by st... [10:34:42] 06Data-Engineering, 10Gerrit, 06Release-Engineering-Team: Reduce size of analytics/superset/deploy.git Gerrit repo - https://phabricator.wikimedia.org/T406856#11318115 (10hashar) 05Open→03Resolved a:03hashar I have deleted the repository which freed some disk space and will make operating Gerrit //... [13:22:37] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07Essential-Work, 13Patch-For-Review: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11318606 (10Gehel) Code merged. Manual test on `an-worker1161`: ` gehel@an-worker1161:~$ sudo nice io... [13:47:01] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list October 2025 - https://phabricator.wikimedia.org/T408561 (10GFontenelle_WMF) 03NEW [14:53:40] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11319096 (10TJones) > I believe we should find what is causing them to... [14:54:56] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06DBA, and 2 others: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587#11319112 (10Daimona) Thank you! ⭐⭐⭐⭐⭐ [15:13:34] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11319263 (10achou) [15:19:38] 06Data-Engineering: Access for new ML team SRE - https://phabricator.wikimedia.org/T408579 (10DPogorzelski-WMF) 03NEW [15:20:24] 06Data-Engineering: Access for new ML team SRE - https://phabricator.wikimedia.org/T408579#11319307 (10DPogorzelski-WMF) [15:29:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11319360 (10dcausse) >>! In T407514#11319096, @TJones wrote: >> I belie... [16:51:42] !log deploying AQS edit-analytics service to pick up edits/per_editor endpoint - T403660 [16:51:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:51:47] T403660: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660 [16:57:11] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10RESTBase-API, 06serviceops, 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11320121 (10Ottomata) I just deployed edit-analytics with the metrics/v3/edits/per_editor endpo... [17:12:49] (03PS1) 10Aqu: Maintain Spark schema after Hive DDL operations [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199473 (https://phabricator.wikimedia.org/T307040) [17:14:27] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform: Add CI step to event schema repositories to test to fail if a schema is deleted - https://phabricator.wikimedia.org/T377023#11320240 (10Ottomata) @JMonton-WMF thanks for the patch! Q: did you explore the possibility of adding this... [17:18:09] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform: Add CI step to event schema repositories to test to fail if a schema is deleted - https://phabricator.wikimedia.org/T377023#11320254 (10JMonton-WMF) I didn't know how that worked. Let me review it and I'll try to move the test there... [17:22:05] (03PS2) 10Aqu: Maintain Spark schema after Hive DDL operations [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199473 (https://phabricator.wikimedia.org/T307040) [17:23:25] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Observability-Alerting, 10Event-Platform: EventgateProduceRateStop / EventGateProduceRateAnomaly alert should be active datacenter aware - https://phabricator.wikimedia.org/T405952#11320273 (10tchin) Talked with Andrew about this more. The main p... [17:27:26] (03PS3) 10Aqu: Maintain Spark schema after Hive DDL operations [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199473 (https://phabricator.wikimedia.org/T307040) [17:32:44] (03PS4) 10Aqu: Maintain Spark schema after Hive DDL operations [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199473 (https://phabricator.wikimedia.org/T307040) [17:33:53] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform: Add CI step to event schema repositories to test to fail if a schema is deleted - https://phabricator.wikimedia.org/T377023#11320334 (10Ottomata) Ya exactly. E.g. https://gitlab.wikimedia.org/repos/data-engineering/schemas-event... [17:45:19] (03CR) 10Ottomata: Maintain Spark schema after Hive DDL operations (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199473 (https://phabricator.wikimedia.org/T307040) (owner: 10Aqu) [17:46:23] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables and into DataHub - https://phabricator.wikimedia.org/T307040#11320421 (10Ottomata) @aqu why mergeComments vs e.g mergeFieldsMetadata like... [18:00:27] (03PS12) 10Ottomata: Add HQL for edit_per_editor_per_page_daily and pageview_per_editor_per_page_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [18:17:42] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11320545 (10Ottomata) @Dbrant! Grea... [18:32:24] (03PS1) 10Snwachukwu: Add Data quality check for Pageview Human-Bot ratio anomaly [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199485 (https://phabricator.wikimedia.org/T407239) [18:34:13] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11320625 (10Michael) >>! In T401021#11313785... [18:38:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10RESTBase-API, 06serviceops, 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11320671 (10HCoplin-WMF) Hey -- sorry for the delay on my side! Still getting caught up on a fe... [18:40:03] (03CR) 10CI reject: [V:04-1] Add Data quality check for Pageview Human-Bot ratio anomaly [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199485 (https://phabricator.wikimedia.org/T407239) (owner: 10Snwachukwu) [18:57:04] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10RESTBase-API, 06serviceops, 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11320731 (10Ottomata) > could you confirm that the version change is going to be applied for al... [18:58:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list October 2025 - https://phabricator.wikimedia.org/T408561#11320733 (10mforns) [18:59:27] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11320735 (10Ottomata) user_central_id is now in Druid mediawiki_history_reduced! Thanks @amastil... [18:59:33] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11320736 (10Ottomata) [19:01:22] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Growth-Team, 10MediaWiki-Page-derived-data, 06Wikipedia-Android-App-Backlog, and 2 others: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics - https://phabricator.wikimedia.org/T403660#11320752 (10Ottomata) Docs are live!... [19:02:02] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11320756 (10achou) I think the key questions... [19:17:01] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10RESTBase-API, 06serviceops, 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11320810 (10HCoplin-WMF) > Even though these are brand new, we wanted to version them at the s... [19:30:14] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10RESTBase-API, 06serviceops, 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11320883 (10Ottomata) > each type of metric is basically its own microservice Which begs the qu... [20:04:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11320956 (10TJones) I often open the target folder to find the jar so I... [20:10:03] (03PS2) 10Snwachukwu: Add Data quality check for Pageview Human-Bot ratio anomaly [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1199485 (https://phabricator.wikimedia.org/T407239) [20:18:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11321018 (10achou) >>! In T401021#11313270,... [20:20:22] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: mediawiki_history_reduced - add page_id and user_central_id fields - https://phabricator.wikimedia.org/T406263#11321042 (10amastilovic) [20:25:13] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: mediawiki_history_reduced - add page_id and user_central_id fields - https://phabricator.wikimedia.org/T406263#11321068 (10amastilovic) [20:26:26] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: mediawiki_history_reduced - add page_id and user_central_id fields - https://phabricator.wikimedia.org/T406263#11321074 (10amastilovic) `user_id`, `user_central_id` and `page_id` fields are now available i... [20:51:07] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11321183 (10Ottomata) > IIUC, you're okay wi... [20:57:05] hmmm [20:57:09] addshore@stat1011:~$ mkdir -p /srv/published/datasets/one-off/wikidata/wmde_wikidata_map [20:57:09] mkdir: cannot create directory ‘/srv/published/datasets/one-off’: Permission denied [20:57:17] I wonder if anyone might be able to help me with that? [21:08:44] ping ottomata [21:17:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:22:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:27:53] I made https://phabricator.wikimedia.org/T408641 for now hoping someone elsewhere might spot it [21:27:58] 06Data-Engineering, 06SRE: stat1011: cannot create directory ‘/srv/published/datasets/one-off’: Permission denied - https://phabricator.wikimedia.org/T408641 (10Addshore) 03NEW [21:52:57] well, i scped them over to stat1010 where i can publish them from for now :) that got me unblocked! [23:41:09] (03PS1) 10Zabe: Introduce MostCategories.hql [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1199521 (https://phabricator.wikimedia.org/T309738)