[04:22:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:47:14] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:11:24] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Apply ppv_package, pfn_name patches in production - https://phabricator.wikimedia.org/T439661#12394159 (10Marostegui) a:03Marostegui [05:13:57] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Apply ppv_package, pfn_name patches in production - https://phabricator.wikimedia.org/T439661#12394160 (10Marostegui) 05Open→03Resolved Done ` cumin2024@db1220.eqiad.wmnet[wikishared]> select count(*) from produnt... [05:56:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:21:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:07:03] 14Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12394216 (10JAllemandou) I have reduced the replication factor for the netflow dataset to 1 instead of 2. Thi... [09:33:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:00:57] 06Data-Engineering: Decide and implement default Hive location for dbt marts - https://phabricator.wikimedia.org/T440170 (10GGoncalves-WMF) 03NEW [10:01:21] 06Data-Engineering: Decide and implement default Hive location for dbt marts - https://phabricator.wikimedia.org/T440170#12394849 (10GGoncalves-WMF) [10:01:22] 10Data-Engineering-Roadmap, 07Epic: Epic: dbt Backlog - https://phabricator.wikimedia.org/T437619#12394850 (10GGoncalves-WMF) [10:02:36] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12394860 (10GGoncalves-WMF) > Some of the other paths that are listed in the table: /user/hive/warehouse/wmf_contributors.db and... [10:03:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:33:44] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop wbc_entity_usage from local clusters - https://phabricator.wikimedia.org/T440173 (10Zabe) 03NEW [11:22:14] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop wbc_entity_usage from local clusters - https://phabricator.wikimedia.org/T440173#12395110 (10Marostegui) @Zabe do you have an estimation on when all of them would be ready? I think it is probably easier to do all at once rather than do one and ke... [11:28:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:03:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:53:10] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop wbc_entity_usage from local clusters - https://phabricator.wikimedia.org/T440173#12395407 (10Zabe) 2-3 weeks for everything except commonswiki. commonswiki itself will take another 3-4 months after that. [13:15:44] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop wbc_entity_usage from local clusters - https://phabricator.wikimedia.org/T440173#12395513 (10Marostegui) p:05Triage→03Medium Let's wait for everything except commons and we can leave that for the very end. Thank you! [13:17:00] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop wbc_entity_usage from local clusters - https://phabricator.wikimedia.org/T440173#12395517 (10Zabe) Sounds good, thanks! [14:23:06] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Emit comprehensive mediawiki user block change information in an event stream - https://phabricator.wikimedia.org/T424685#12395932 (10JMonton-WMF) a:05Ottomata→03JMonton-WMF [14:24:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [14:38:11] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12396037 (10BTullis) This now works. Create a token: ` root@deploy1003:~# kubectl -n analytics-test create token... [14:49:06] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 06SRE: Reduce Kerberos logs produced by Presto - https://phabricator.wikimedia.org/T353802#12396090 (10LSobanski) [14:52:07] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12396099 (10BTullis) [14:59:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [15:40:54] (03CR) 10Xcollazo: Add spark operational metrics derived from event logs (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [16:34:01] 06Data-Engineering: Add width bands to the mediarequests API - https://phabricator.wikimedia.org/T439420#12396702 (10Aklapper) [17:00:12] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement 7z compression for v2 history dumps - https://phabricator.wikimedia.org/T437454#12396782 (10xcollazo) 851 / 859 7z tasks have been successful l as of this comment. the remaining are just part of the long tail of huge wikis. I've spot checked s... [17:01:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:18:13] 06Data-Engineering: ruwiki History File Export has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219 (10xcollazo) 03NEW [17:18:22] 06Data-Engineering: ruwiki History File Export has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219#12396912 (10xcollazo) [17:18:24] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07Epic: XML Dumps v1 sunset - https://phabricator.wikimedia.org/T437518#12396913 (10xcollazo) [17:18:49] 06Data-Engineering: Current ruwiki History File Export run has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219#12396915 (10xcollazo) [17:20:07] 06Data-Engineering: Current ruwiki History File Export run has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219#12396928 (10xcollazo) [17:20:54] 06Data-Engineering: Current ruwiki History File Export run has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219#12396931 (10xcollazo) [17:31:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:14:36] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Current ruwiki History File Export run has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219#12397194 (10xcollazo) [18:35:49] 06Data-Engineering, 06Data-Platform-SRE: Remove the spark_shuffle_3_3 and spark_shuffle_3_4 YARN aux services - https://phabricator.wikimedia.org/T440229 (10xcollazo) 03NEW [18:36:27] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Current ruwiki History File Export run has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219#12397308 (10xcollazo) **Summary** The ruwiki export fails because the auth handshake to the YARN... [18:55:46] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): html enrichment pipeline should record bytes instead of characters - https://phabricator.wikimedia.org/T440233 (10AKhatun_WMF) 03NEW [19:21:04] 06Data-Engineering, 10Dumps-Generation, 10MediaWiki-Core-Snapshots: Importing the same page twice creates duplicate revisions when its text contains \r\n - https://phabricator.wikimedia.org/T440235 (10Physikerwelt) 03NEW [19:33:07] 06Data-Engineering, 10Event-Platform, 07good first task: EventBus - move event producers to a different extension - https://phabricator.wikimedia.org/T432050#12397614 (10Ottomata) @mpopov @phuedx btw, for T436695+T432604 (and soon for T438491), we will be adding some 'event data product' producer code to Wik... [19:50:00] 06Data-Engineering: Revisit thresholds for the Human-Bot Page View Count Anomaly alert - https://phabricator.wikimedia.org/T438957#12397723 (10dr0ptp4kt) Anecdata: I'm on Ops Week and am seeing the same sort of thing continuing here in October 2026. I'll bump the IM thread about this. [20:08:15] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Current ruwiki History File Export run has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219#12397800 (10xcollazo) `ruwiki` just succeeded on its 3rd attempt. These ch... [20:09:13] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Current ruwiki History File Export run has failed 2 times, looks like it has struggled in the last couple months - https://phabricator.wikimedia.org/T440219#12397802 (10xcollazo) 05Open→03In progress p:05Triage→03Medium a:0... [21:02:44] 06Data-Engineering, 10Event-Platform, 07good first task: EventBus - move event producers to a different extension - https://phabricator.wikimedia.org/T432050#12398118 (10mpopov) No issues with `EventDataProducts` from me. [21:14:26] (03PS2) 10Kengkong: Added MapChart population per capita for editors and viewers [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1346120 (https://phabricator.wikimedia.org/T424227) [21:35:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:49:40] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Backfill and update schedule of gcp_gsc_export.max_date_sitedata_daily_aggregate - https://phabricator.wikimedia.org/T436999#12398196 (10nshahquinn-wmf) Status note: the backfill of the new table (`wmf_readership.google_site... [21:49:51] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement default expiration for staging and intermediate dbt models - https://phabricator.wikimedia.org/T439690#12398197 (10Mayakp.wiki) hey @GGoncalves-WMF , We would appreciate getting some time to think about the impact of this change. Ive added it t... [21:55:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:56:10] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Non-incremental active UWERs job has incorrect Airflow schedule and dependencies - https://phabricator.wikimedia.org/T438584#12398203 (10nshahquinn-wmf) @amastilovic I've merged both MRs, but something is not right with [the... [22:33:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:35:52] 06Data-Engineering, 10Event-Platform: Seeing frequent 504 timeouts from the event service lose api-request events, erroring with "Invalid event service response body" - https://phabricator.wikimedia.org/T440247 (10Jdforrester-WMF) 03NEW [22:54:24] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement default expiration for staging and intermediate dbt models - https://phabricator.wikimedia.org/T439690#12398343 (10nshahquinn-wmf) To provide a little more detail on what @Mayakp.wiki said, we (the Movement Insights team) aren't ready for this... [23:08:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:28:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt-jobs backfill: second-week retention metric - https://phabricator.wikimedia.org/T440249 (10Mayakp.wiki) 03NEW [23:53:54] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt-jobs backfill: constructive_activation_and_retention_monthly - https://phabricator.wikimedia.org/T440251#12398482 (10Mayakp.wiki)