[00:12:55] <wikibugs>	 (03CR) 10Dr0ptp4kt: [C:03+1] "LGTM. Reference: https://gitlab.wikimedia.org/-/snippets/313" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352273 (https://phabricator.wikimedia.org/T436999) (owner: 10Aqu)
[01:49:19] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06MediaWiki-API-Platform-Team, 07OKR-Work: REST API vNext: Create dedicated REST shared-schema repository - https://phabricator.wikimedia.org/T432799#12404411 (10KineticPelagic)
[02:18:06] <jinxer-wm>	 FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[02:33:06] <jinxer-wm>	 RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[03:55:06] <jinxer-wm>	 FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[04:05:06] <jinxer-wm>	 RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[04:22:09] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06Privacy Engineering, 06Security-Team, and 3 others: Privacy review of x1 tables in preparation of adding them to wikireplicas - https://phabricator.wikimedia.org/T415219#12404592 (10Ladsgroup) Gentle ping!
[04:45:18] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10MediaWiki-extensions-Wikibase-Client, 10Wikidata: Move the wbc_entity_usage table onto a dedicated DB shard - https://phabricator.wikimedia.org/T176273#12404618 (10Ladsgroup) →14Duplicate dup:03T424133
[07:03:04] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05acl*data-engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12404740 (10RKemper) >>! In T435480#12385842, @atsuko wrote: > Moving this two work i...
[07:18:15] <wikibugs>	 (03CR) 10A-pizzata: [C:03+1] Remove unused bin/hdfs-rsync script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352228 (https://phabricator.wikimedia.org/T440383) (owner: 10Xcollazo)
[08:52:34] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06MediaWiki-Media-Platform-Team, 06ServiceOps, and 3 others: DC Switchover - MW requests to EventGate cause long queuing waiting time - https://phabricator.wikimedia.org/T439552#12405037 (10MLechvien-WMF) 05Open→03Resolved Based on the last discussion,...
[08:53:57] <wikibugs>	 (03CR) 10Aqu: [C:03+2] Fix Iceberg Bug for wmf_readership.google_sitedata_daily_aggregate [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352273 (https://phabricator.wikimedia.org/T436999) (owner: 10Aqu)
[08:54:12] <wikibugs>	 (03CR) 10Aqu: [V:03+2 C:03+2] Fix Iceberg Bug for wmf_readership.google_sitedata_daily_aggregate [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352273 (https://phabricator.wikimedia.org/T436999) (owner: 10Aqu)
[09:02:18] <wikibugs>	 (03CR) 10Joal: [C:03+1] Update partitioning of api_requests_refined to use ts column. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1344135 (https://phabricator.wikimedia.org/T438655) (owner: 10KCVelaga)
[09:13:01] <wikibugs>	 (03CR) 10Joal: [C:03+1] Update partitioning of api_requests_refined to use ts column. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1344135 (https://phabricator.wikimedia.org/T438655) (owner: 10KCVelaga)
[09:22:30] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review, 06Product-Analytics (Kanban): Airflow DAG: refine_api_requests_hourly - use artifact approach for rules.json - https://phabricator.wikimedia.org/T437058#12405200 (10KCVelaga_WMF)
[11:35:15] <wikibugs>	 (03CR) 10Joal: Fix Iceberg Bug for wmf_readership.google_sitedata_daily_aggregate (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352273 (https://phabricator.wikimedia.org/T436999) (owner: 10Aqu)
[11:37:47] <wikibugs>	 (03CR) 10Joal: [C:03+1] Remove unused bin/hdfs-rsync script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352228 (https://phabricator.wikimedia.org/T440383) (owner: 10Xcollazo)
[12:16:27] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Apply Iceberg table maintenance to all dbt marts - https://phabricator.wikimedia.org/T439692#12405800 (10JAllemandou) I'm in agreement with the almost every comment @xcollazo has made above. The only point I'd like to discuss is data deletion vs iceberg...
[12:16:33] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Load Google Search Console data from BigQuery as soon as it's available and reload when data corrections occur - https://phabricator.wikimedia.org/T436193#12405801 (10Antoine_Quhen) 05Resolved→03In progress 2 bugs appeared at fi...
[13:13:09] <wikibugs>	 (03CR) 10Xcollazo: [C:03+2] Remove unused bin/hdfs-rsync script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352228 (https://phabricator.wikimedia.org/T440383) (owner: 10Xcollazo)
[13:13:26] <wikibugs>	 (03CR) 10Xcollazo: [V:03+2 C:03+2] Remove unused bin/hdfs-rsync script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1352228 (https://phabricator.wikimedia.org/T440383) (owner: 10Xcollazo)
[13:28:06] <jinxer-wm>	 FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[13:55:18] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06SRE Observability, 13Patch-For-Review: Finalize and/or standardize metrics - https://phabricator.wikimedia.org/T438383#12406301 (10herron) In terms of aligning with existing metric names, in this case I'd say the inverse of the proposed naming is t...
[14:03:06] <jinxer-wm>	 RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[14:07:50] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06SRE Observability, 13Patch-For-Review: Finalize and/or standardize metrics - https://phabricator.wikimedia.org/T438383#12406381 (10Ottomata) Oh okay interesting!  @herron I was imagining that efforts like {T437630} and {T436785} https://gitlab.wiki...
[14:12:47] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Update legacy dumps HTML to reflect sunsetting date - https://phabricator.wikimedia.org/T440466 (10xcollazo) 03NEW
[14:13:08] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Update legacy dumps HTML to reflect sunsetting date - https://phabricator.wikimedia.org/T440466#12406426 (10xcollazo)
[14:13:09] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07Epic: XML Dumps v1 sunset - https://phabricator.wikimedia.org/T437518#12406427 (10xcollazo)
[14:39:44] <wikibugs>	 06Data-Engineering, 10Dumps-Generation: Set NameWidth=* for dump listings - https://phabricator.wikimedia.org/T435849#12406626 (10xcollazo) This should be an easy quality of life fix, will do as part of a current stream of work in this area.
[14:40:16] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Update legacy dumps HTML to reflect sunsetting date - https://phabricator.wikimedia.org/T440466#12406637 (10xcollazo) Let's do {T435849} at the same time.
[14:53:52] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Connection-Team, 06Growth-Team, 10Thanks: Define the requirements for Thanks instrumentation - https://phabricator.wikimedia.org/T438493#12406765 (10Snwachukwu) Per our policy for producing events for non-internal data products, we would not use...
[15:05:11] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: User and team Hive database cleanup tracking for Fall 2026 - https://phabricator.wikimedia.org/T437733#12406879 (10KCVelaga_WMF) @xcollazo per our discussion, the following staging and intermediate tables can be cleaned up.   ` wmf_...
[15:07:46] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Backfill and update schedule of gcp_gsc_export.max_date_sitedata_daily_aggregate - https://phabricator.wikimedia.org/T436999#12406904 (10Antoine_Quhen) The pipeline looks fine now.  In my study of the Spark/Iceberg bug, I no...
[15:08:15] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review: [Data Persistence Design Review] Thanks - https://phabricator.wikimedia.org/T440474#12406907 (10Snwachukwu)
[15:33:12] <jinxer-wm>	 FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[15:36:44] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Remove the spark_shuffle_3_3 and spark_shuffle_3_4 YARN aux services - https://phabricator.wikimedia.org/T440229#12407119 (10Gehel)
[15:37:23] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Platform, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12407133 (10Gehel)
[15:38:44] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Remove the spark_shuffle_3_3 and spark_shuffle_3_4 YARN aux services - https://phabricator.wikimedia.org/T440229#12407160 (10Gehel) p:05Triage→03Medium
[15:40:19] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE, 10Test Kitchen, 07Essential-Work: Get schema browser in GrowthBook to work - https://phabricator.wikimedia.org/T440403#12407225 (10Gehel) p:05Triage→03Medium
[15:48:49] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review: [Data Persistence Design Review] Thanks - https://phabricator.wikimedia.org/T440474#12407322 (10Snwachukwu)
[15:50:31] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review: [Data Persistence Design Review] Thanks - https://phabricator.wikimedia.org/T440474#12407328 (10Snwachukwu)
[15:59:38] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12407410 (10BTullis)
[16:01:26] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12407434 (10BTullis) Moving under {T439657} because I think that some of the same fixes for that can be applied to `conda-analytics` and wi...
[16:07:23] <wikibugs>	 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments stuck with very outdated packages - https://phabricator.wikimedia.org/T423052#12407493 (10BTullis)
[16:07:57] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE: Conda-Analytics has package conflict when trying to install R with key packages (R-Arrow and R-Stringi) - https://phabricator.wikimedia.org/T391911#12407500 (10BTullis)
[16:09:41] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE, 07Epic, 13Patch-For-Review: Upgrade to Spark 3.5 - https://phabricator.wikimedia.org/T338057#12407526 (10BTullis)
[16:10:40] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE: Upgrade to JupyterLab ≥ 4.2 - https://phabricator.wikimedia.org/T391905#12407535 (10BTullis)
[16:13:12] <jinxer-wm>	 RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly
[16:14:30] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE, 07Epic, 13Patch-For-Review: Upgrade to Spark 3.5 - https://phabricator.wikimedia.org/T338057#12407621 (10BTullis)
[16:28:29] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06SRE Observability, 13Patch-For-Review: Finalize and/or standardize metrics - https://phabricator.wikimedia.org/T438383#12407853 (10herron) Re: automating dashboards, Grafana can approach this today, for example using a query to the effect of `/^(.+...
[16:35:24] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review: [Data Persistence Design Review] Thanks - https://phabricator.wikimedia.org/T440474#12407903 (10Snwachukwu)
[16:35:29] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06SRE Observability, 13Patch-For-Review: Finalize and/or standardize metrics - https://phabricator.wikimedia.org/T438383#12407908 (10Ottomata) +1 ty!
[16:37:01] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06ServiceOps, 10ServiceOps-SharedInfra, 13Patch-For-Review: Create namespace for LAC lambdas - https://phabricator.wikimedia.org/T439879#12407923 (10Ottomata)
[16:39:55] <wikibugs>	 06Data-Engineering, 10Dumps-Generation: Set NameWidth=* for dump listings - https://phabricator.wikimedia.org/T435849#12407932 (10xcollazo) >>! In T435849#12406626, @xcollazo wrote: > This should be an easy quality of life fix, will do as part of a current stream of work in this area.  Actually I take it back,...
[16:41:00] <wikibugs>	 06Data-Engineering, 10Dumps-Generation: Set NameWidth=* for dump listings - https://phabricator.wikimedia.org/T435849#12407934 (10xcollazo) 05Open→03Declined @Etrapani we are not moving forward with this change today, but thank you for your report. Hopefully one of the listed workaround works for you.
[16:41:02] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Dashiki: Pingback dashboard missing values after January 2026 - https://phabricator.wikimedia.org/T440089#12407939 (10Ottomata) Related: T415283 & T418190
[16:43:17] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Update legacy dumps HTML to reflect sunsetting date - https://phabricator.wikimedia.org/T440466#12407951 (10xcollazo) The sunset notice is in three places. Each place has a patch:  - https://gerrit.wikimedia.org/r/1352551 (`operatio...
[16:43:53] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Update legacy dumps HTML to reflect sunsetting date - https://phabricator.wikimedia.org/T440466#12407954 (10xcollazo) >>! In T440466#12406637, @xcollazo wrote: > Let's do {T435849} at the same time.  Decided to not do this. Details...
[17:03:18] <wikibugs>	 10Data-Engineering-Roadmap, 07Epic: [Epic] KAPOW: The next generation of bot detection in the Data Platform. - https://phabricator.wikimedia.org/T425661#12408098 (10mforns)
[20:04:22] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06MediaWiki-API-Platform-Team, 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 - adapt page change kind model to MediaWiki's PageUpdateCauses. - https://phabricator.wikimedia.org/T430588#12408926 (10Ottomata) Okay, my 'friend' and I...
[20:08:39] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06SRE Observability, 13Patch-For-Review: Finalize and/or standardize metrics - https://phabricator.wikimedia.org/T438383#12408944 (10tchin) BTW NodeJS's prometheus client exports default metrics, anything that uses it (including service-utils) alread...
[20:20:14] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review, 06Product-Analytics (Kanban): Update partitioning of api_requests_refined to use ts col - https://phabricator.wikimedia.org/T438655#12408972 (10dr0ptp4kt) Thanks @xcollazo.  Update from https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1...
[21:47:37] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt-jobs backfill: constructive_activation_and_retention_monthly - https://phabricator.wikimedia.org/T440251#12409197 (10amastilovic)
[21:47:48] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Movement-Insights (FY26-27 H1): dbt-jobs backfill: constructive_activation_and_retention_monthly - https://phabricator.wikimedia.org/T440251#12409198 (10amastilovic) Task completed.
[22:21:36] <wikibugs>	 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06SRE Observability, 13Patch-For-Review: Finalize and/or standardize metrics - https://phabricator.wikimedia.org/T438383#12409245 (10AKhatun_WMF) Yeah, I had checked service utils code. It has interceptors that will basically emit metric on calls wit...