[02:29:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:04:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:41:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:26:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:04:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12399280 (10APizzata-WMF) [08:29:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:38:01] 06Data-Engineering, 10Event-Platform, 07good first task: EventBus - move event producers to a different extension - https://phabricator.wikimedia.org/T432050#12399424 (10phuedx) Same here. `EventDataProducts` is descriptive without being overly constraining. [08:44:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:28:18] 06Data-Engineering: Revisit thresholds for the Human-Bot Page View Count Anomaly alert - https://phabricator.wikimedia.org/T438957#12399615 (10GGoncalves-WMF) Note: This is based on the legacy bot detection still, not KAPOW. We do need to revisit the alert. If I understand correctly, the conditions causing the... [09:32:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:33:39] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12399635 (10APizzata-WMF) [09:35:46] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop wbc_entity_usage from local clusters - https://phabricator.wikimedia.org/T440173#12399643 (10Ladsgroup) Something to note: This table is quite large in some sections. s4 is the biggest with >200GB but in s6 it's 26GB: ` root@db1168:/srv/sqldata#... [09:37:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:52:18] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Implement default expiration for staging and intermediate dbt models - https://phabricator.wikimedia.org/T439690#12399921 (10GGoncalves-WMF) Thanks for the context, and yes, happy to discuss tomorrow! Indeed, if we "just" add an expiration policy which t... [11:22:22] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12400031 (10APizzata-WMF) Merged all the base models and waiting for tomorrow sqoop and run to start the back fill.... [12:42:40] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Classify requests to developer.wikimedia.org as pageviews - https://phabricator.wikimedia.org/T428550#12400341 (10GGoncalves-WMF) T434137 is adding instrumentation to the Developer Portal for measuring pageviews, separately from the existing pageviews pi... [13:25:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:55:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [14:06:25] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Apply Iceberg table maintenance to all dbt marts - https://phabricator.wikimedia.org/T439692#12401038 (10xcollazo) Some comments on the approach below: >I considered scoping the scan to a fixed list of dbt-managed schemas, but that list would need to st... [14:06:54] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Apply Iceberg table maintenance to all dbt marts - https://phabricator.wikimedia.org/T439692#12401042 (10xcollazo) CC @JAllemandou for if he has time to review this proposal as well. [14:23:34] (03CR) 10Milimetric: "Adding Marcel in case he hasn't seen this. I'd think at this point most of this was discussed/incorporated into Webrequest v2 / KAPOW?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1196959 (owner: 10CDanis) [14:32:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [14:37:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [15:28:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:13:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:30:30] 06Data-Engineering, 10Event-Platform: Seeing frequent 504 timeouts from the event service lose api-request events, erroring with "Invalid event service response body" - https://phabricator.wikimedia.org/T440247#12402458 (10Ottomata) Hm, your time range is last few months, so this might just be noise, but: T439... [16:31:51] 06Data-Engineering, 10Event-Platform: Seeing frequent 504 timeouts from the event service lose api-request events, erroring with "Invalid event service response body" - https://phabricator.wikimedia.org/T440247#12402473 (10Ottomata) > the timeout has been lowered to 11s in the service mesh Oh, nevermind. That... [16:37:50] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12402505 (10nshahquinn-wmf) Thank you @APizzata-WMF! @CMyrick-WMF has a small change that she wants to make to the... [17:22:58] 06Data-Engineering, 13Patch-For-Review: Revisit thresholds for the Human-Bot Page View Count Anomaly alert - https://phabricator.wikimedia.org/T438957#12402807 (10dr0ptp4kt) >>! In T438957#12399613, @GGoncalves-WMF wrote: > In the meantime, I'm perfectly fine for us to stop spending Ops Week time dealing with... [17:35:02] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12402862 (10nshahquinn-wmf) Okay, I've merged @CMyrick-WMF's [MR](https://gitlab.wikimedia.org/repos/data-engineeri... [17:58:37] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt-jobs backfill: constructive_activation_and_retention_monthly - https://phabricator.wikimedia.org/T440251#12402984 (10Mayakp.wiki) [18:09:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): hdfs-rsync: Skip incomplete source directories and tolerate vanished source files - https://phabricator.wikimedia.org/T440383 (10xcollazo) 03NEW [18:12:27] (03PS1) 10Xcollazo: Remove unused bin/hdfs-rsync script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352228 (https://phabricator.wikimedia.org/T440383) [18:29:37] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10Kafka-Infrastructure: beta - Run a single event platform Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12403106 (10tchin) Peeking into the data folder, there's no new activity in any of the topic partitions since Sept 17th, with the onl... [18:43:08] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10Kafka-Infrastructure: beta - Run a single event platform Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12403139 (10tchin) a:03tchin [18:43:17] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10Kafka-Infrastructure: beta - Run a single event platform Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12403140 (10tchin) Cleaned up `deployment-kafka-jumbo-5`, `deployment-kafka-jumbo-8`, `and deployment-kafka-jumbo-9` instances using... [18:44:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Cloud-VPS (Debian Bullseye Deprecation), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12403144 (10tchin) All instances have been migrated now [18:45:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Cloud-VPS (Debian Bullseye Deprecation), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12403146 (10Ottomata) 🎉 [18:51:37] (03CR) 10Dr0ptp4kt: "Holding on `analytics/refinery deploy` for the time being. Hoping to clarify order of operations and if we should hold onto an existing co" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1344135 (https://phabricator.wikimedia.org/T438655) (owner: 10KCVelaga) [19:15:38] 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review, 06Product-Analytics (Kanban): Update partitioning of api_requests_refined to use ts col - https://phabricator.wikimedia.org/T438655#12403276 (10dr0ptp4kt) https://wikitech.wikimedia.org/w/index.php?title=Data_Platform_Engineering%2FOps_week%... [19:16:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: hdfs-rsync: Skip incomplete source directories and tolerate vanished source files - https://phabricator.wikimedia.org/T440383#12403277 (10xcollazo) [19:22:46] 06Data-Engineering, 10Event-Platform, 07good first task: EventBus - move event producers to a different extension - https://phabricator.wikimedia.org/T432050#12403304 (10Anushka10patel) a:05Anushka10patel→03None [20:01:24] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07Epic: XML Dumps v1 sunset - https://phabricator.wikimedia.org/T437518#12403433 (10xcollazo) [20:01:25] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: hdfs-rsync: Skip incomplete source directories and tolerate vanished source files - https://phabricator.wikimedia.org/T440383#12403432 (10xcollazo) [20:01:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: hdfs-rsync: Skip incomplete source directories - https://phabricator.wikimedia.org/T440383#12403434 (10xcollazo) [20:26:58] (03PS1) 10Aqu: Fix Iceberg Bug for wmf_readership.google_sitedata_daily_aggregate [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352273 (https://phabricator.wikimedia.org/T436999) [20:31:47] (03CR) 10Xcollazo: Update partitioning of api_requests_refined to use ts column. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1344135 (https://phabricator.wikimedia.org/T438655) (owner: 10KCVelaga) [20:36:46] (03CR) 10Ottomata: [C:03+1] Remove unused bin/hdfs-rsync script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352228 (https://phabricator.wikimedia.org/T440383) (owner: 10Xcollazo) [20:38:04] 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review, 06Product-Analytics (Kanban): Update partitioning of api_requests_refined to use ts col - https://phabricator.wikimedia.org/T438655#12403622 (10xcollazo) > @amastilovic @xcollazo any other actions you can think that should immediately preced... [21:27:22] 06Data-Engineering, 06Data-Platform-SRE, 10Test Kitchen, 07Essential-Work: Get schema browser in GrowthBook to work - https://phabricator.wikimedia.org/T440403 (10mpopov) 03NEW [21:43:25] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 10EditCheck, 10VisualEditor, and 3 others: Implement sampling requirements for Headless VE technical pilot - https://phabricator.wikimedia.org/T433024#12403893 (10ppelberg) [21:43:47] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 10EditCheck, 10VisualEditor, and 3 others: Implement sampling requirements for Headless VE technical pilot - https://phabricator.wikimedia.org/T433024#12403894 (10ppelberg) Updating scaling schedule based on decisions made during t... [22:02:59] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Non-incremental active UWERs job has incorrect Airflow schedule and dependencies - https://phabricator.wikimedia.org/T438584#12404004 (10nshahquinn-wmf) @amastilovic fixed it by [increasing the job's Spark driver memory](htt... [22:19:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Backfill and update schedule of gcp_gsc_export.max_date_sitedata_daily_aggregate - https://phabricator.wikimedia.org/T436999#12404038 (10nshahquinn-wmf) @Antoine_Quhen fixed the bug and rebackfilled the... [22:19:49] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Backfill and update schedule of gcp_gsc_export.max_date_sitedata_daily_aggregate - https://phabricator.wikimedia.org/T436999#12404042 (10nshahquinn-wmf) Also @Antoine_Quhen can you configure the new tab... [23:09:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:17:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Non-incremental active UWERs job has incorrect Airflow schedule and dependencies - https://phabricator.wikimedia.org/T438584#12404205 (10amastilovic) Incorrect data rows were deleted. [23:33:47] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Non-incremental active UWERs job has incorrect Airflow schedule and dependencies - https://phabricator.wikimedia.org/T438584#12404261 (10nshahquinn-wmf) @AMastilovic also restarted the Datalake-to-Postgres job to get the bac... [23:49:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly