[01:30:53] (03PS1) 10Gerrit maintenance bot: Add min.wikisource to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1198722 (https://phabricator.wikimedia.org/T408343) [01:32:12] (03PS1) 10Gerrit maintenance bot: Add pcm.wikiquote to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1198724 (https://phabricator.wikimedia.org/T408351) [05:52:43] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Observability-Alerting, 10Event-Platform: EventgateProduceRateStop / EventGateProduceRateAnomaly alert should be active datacenter aware - https://phabricator.wikimedia.org/T405952#11310539 (10tchin) Wow, this was harder than I thought. So what w... [06:08:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:23:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:04:23] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11311125 (10fgiunchedi) p:05Triage→03Medium [11:38:47] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform: Add CI step to event schema repositories to test to fail if a schema is deleted - https://phabricator.wikimedia.org/T377023#11311435 (10JMonton-WMF) a:03JMonton-WMF [13:00:59] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11311664 (10pfischer) [13:01:23] 06Data-Engineering, 10CirrusSearch, 06Data-Platform-SRE, 10DPE-Mediawiki-Content, and 3 others: Source the CirrusSearch index dumps from hadoop instead of a MW maintenance script - https://phabricator.wikimedia.org/T366248#11311662 (10pfischer) [13:01:35] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 06Discovery-Search (2025.10.20 - 2025.11.07), and 2 others: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies ... - https://phabricator.wikimedia.org/T367405#11311666 [13:03:22] 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 06Discovery-Search (2025.10.20 - 2025.11.07), and 2 others: Eventutilities Flink: port SerDe tests from SUP - https://phabricator.wikimedia.org/T404597#11311704 (10pfischer) [13:09:13] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Product-Analytics, 10Event-Platform: [Event Platform] Disable default collection of user agent for analytics streams - https://phabricator.wikimedia.org/T384964#11311727 (10Ottomata) In Slack, @JMonton-WMF wrote: > I could disable the collection... [13:12:28] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Product-Analytics, 10Event-Platform: [Event Platform] Disable default collection of user agent for analytics streams - https://phabricator.wikimedia.org/T384964#11311744 (10Ottomata) Good qs! Hm. I think: > Should I enable User Agent in all even... [13:13:31] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Product-Analytics, 10Event-Platform: [Event Platform] Disable default collection of user agent for analytics streams - https://phabricator.wikimedia.org/T384964#11311750 (10Ottomata) > Should I try to follow each event to find if we end up using... [14:10:31] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Traffic: Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360#11311915 (10Ottomata) From an-launcher1003: `lang=bash $ sudo -u analytics kerberos-run-command analytics spark3-sql ` `lang=sql DROP TABLE `event... [14:10:33] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Traffic: Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360#11311916 (10Ottomata) [14:42:01] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10RESTBase-API, 06serviceops, 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11312091 (10Sfaci) > It might be possible to restructure AQS code to make it possible to host m... [14:48:20] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10RESTBase-API, 06serviceops, 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11312131 (10Sfaci) > It might be possible to restructure AQS code to make it possible to host m... [14:48:50] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Global Editor Metrics - Druid mediawiki_history_reduced changes - https://phabricator.wikimedia.org/T406069#11312132 (10Ottomata) Ya! Let's go! [14:52:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06SRE: Move Druid realtime configuration out of Refinery into standalone repo on GitLab - https://phabricator.wikimedia.org/T407994#11312146 (10Ottomata) Hm! Before we do this, should we take a bigger picture look around what we intend to do with re... [14:58:18] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312153 (10Eevans) [15:01:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312156 (10Eevans) [15:02:03] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10RESTBase-API, 06serviceops, 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11312160 (10Ottomata) ...I am more and more leaning towards in the future just implementing API... [15:03:22] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312163 (10Eevans) [15:07:02] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312184 (10Eevans) [15:09:10] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Unable to change input dump path of Airflow commons_structured_data_dump_to_hive_weekly dag - https://phabricator.wikimedia.org/T408400 (10Snwachukwu) 03NEW [15:09:22] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content: Missing/inconsistent page_redirect_target field for redirects in Mediawiki content current v1 dumps - https://phabricator.wikimedia.org/T400632#11312206 (10Antoine_Quhen) [15:09:34] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Unable to change input dump path of Airflow commons_structured_data_dump_to_hive_weekly dag - https://phabricator.wikimedia.org/T408400#11312208 (10Snwachukwu) a:03Snwachukwu [15:10:53] 06Data-Engineering, 06Data-Persistence, 06DBA, 10MediaWiki-Core-Revision-backend, and 6 others: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#11312221 (10Zabe) Looks like dewikivoyage and itwikivoyage are fixed. ` wikiadmin2023@10.192.32.16(dewikivoyage)> select count(*) from content... [15:14:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312246 (10Ottomata) I like it! Some field... [15:15:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Unable to change input dump path of Airflow commons_structured_data_dump_to_hive_weekly dag - https://phabricator.wikimedia.org/T408400#11312248 (10Snwachukwu) [15:16:17] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312271 (10Eevans) [15:16:43] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Vet JA3N data in webrequest and pageview_actor - https://phabricator.wikimedia.org/T408404 (10mforns) 03NEW [15:18:38] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312293 (10achou) > QUESTION: What is Tone... [15:19:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Design and populate temporary table for JA3N analysis - https://phabricator.wikimedia.org/T408405 (10mforns) 03NEW [15:21:16] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Create a new gitlab repository for use with dbt - https://phabricator.wikimedia.org/T406765#11312319 (10Ottomata) I don't have a strong preference other than not `dbt`. [15:22:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Unable to change input dump path of Airflow commons_structured_data_dump_to_hive_weekly dag - https://phabricator.wikimedia.org/T408400#11312322 (10Snwachukwu) I'm using this ticket as an opportunity to perform the following upgrades on the airflow da... [15:23:56] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Create a new gitlab repository for use with dbt - https://phabricator.wikimedia.org/T406765#11312328 (10xcollazo) `this-is-a-repo-that-holds-dbt-stuff-but-not-dbt-itself` ? ... I'll show myself out. [15:24:36] 06Data-Engineering, 06Experimentation Lab, 10MediaWiki-extensions-WikimediaEvents, 05MW-1.45-notes (1.45.0-wmf.24; 2025-10-21): Prepare mediawiki-client-error Logstash dashboards for mobile subdomain sunsetting - https://phabricator.wikimedia.org/T400852#11312338 (10Ottomata) FYI, in case this is usefu... [15:36:21] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312488 (10Eevans) >>! In T401021#11312293,... [15:38:39] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312493 (10Ottomata) > consumes the mediawi... [15:44:34] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312545 (10Ottomata) > But also, it kind of... [16:01:40] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312595 (10Michael) >>! In T401021#11307866... [16:03:54] (03CR) 10Ottomata: [C:03+1] "I didn't review semantics deeply, but let's go and try and revisit anything we have to later. Let's get user_central_id out there!!!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1198607 (https://phabricator.wikimedia.org/T406263) (owner: 10Aleksandar Mastilovic) [16:06:10] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312636 (10Eevans) >>! In T401021#11312545,... [16:10:34] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11312654 (10Eevans) >>! In T401021#11312595,... [16:11:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Create a new gitlab repository for use with dbt - https://phabricator.wikimedia.org/T406765#11312666 (10JMonton-WMF) I have renamed the path and name to `dbt-jobs` (https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs) for now. As we are the on... [18:32:36] (03PS4) 10Ottomata: Add HQL for wmf_contributors.user_edited_pages_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [19:00:05] (03PS5) 10Ottomata: Add HQL for wmf_contributors.user_edited_pages_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [19:04:12] (03PS6) 10Ottomata: Add HQL for wmf_contributors.user_edited_pages_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [19:07:17] (03CR) 10Mforns: [C:03+1] "Left a couple superficial comments" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) (owner: 10Ottomata) [19:17:01] (03CR) 10Ottomata: "I wonder if we should just create this table in the edit/ directory and call it" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) (owner: 10Ottomata) [19:20:57] (03CR) 10Ottomata: Add HQL for wmf_contributors.user_edited_pages_daily (038 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) (owner: 10Ottomata) [19:21:02] (03PS7) 10Ottomata: Add HQL for wmf_contributors.user_edited_pages_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [19:48:11] (03PS8) 10Ottomata: Add HQL for wmf_contributors.user_edited_pages_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [19:48:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work, 13Patch-For-Review: Global Editor Metrics - Data Pipeline - edit_per_user_per_page_daily - https://phabricator.wikimedia.org/T407559#11313588 (10Ottomata) [19:50:02] (03PS9) 10Ottomata: Add HQL for wmf_contributors.user_edited_pages_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [20:04:41] (03PS10) 10Ottomata: Add HQL for wmf_contributors.user_edited_pages_daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [20:27:30] (03PS11) 10Ottomata: Add HQL for wmf_contributors.edit_per_editor_per_page_daily table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196892 (https://phabricator.wikimedia.org/T407559) [20:28:03] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work, 13Patch-For-Review: Global Editor Metrics - Data Pipeline - edit_per_editor_per_page_daily - https://phabricator.wikimedia.org/T407559#11313681 (10Ottomata) [20:43:25] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 07Essential-Work, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11313704 (10Mayakp.wiki) Hey everyone we discusse... [20:58:46] (03PS1) 10Ottomata: Add HQL for wmf_readership.pageview_per_editor_per_page_daily table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1199067 (https://phabricator.wikimedia.org/T405039) [21:07:30] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11313785 (10Eevans) >>! In T401021#11313193,... [23:39:19] 06Data-Engineering, 06Data-Platform-SRE: Improve housekeeping of files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11314723 (10bking) I want to share a bit more context based on [[ https://wikimedia.slack.com/archives/C055QGPTC69/p1761588688501909 | this Slack thread ]] . As @Gehel...