[01:10:49] 06Data-Engineering, 10Wmfdata-Python, 06Movement-Insights (FY25-26 H1): Update Wmfdata developer documentation to correctly cover working with UV - https://phabricator.wikimedia.org/T402990#11422171 (10nshahquinn-wmf) [03:28:34] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.8 placeholder sub-ticket - https://phabricator.wikimedia.org/T411454 (10tchin) 03NEW [03:53:20] (03CR) 10Aleksandar Mastilovic: [C:03+2] Add HQL for pageviews_per_editor and pageview_top_pages_per_editor [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [03:53:24] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Add HQL for pageviews_per_editor and pageview_top_pages_per_editor [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [06:31:29] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11422603 (10Marostegui) [06:31:59] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11422604 (10Marostegui) [06:47:09] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11422607 (10Marostegui) [06:56:49] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11422610 (10Marostegui) [06:56:51] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11422611 (10Marostegui) [08:55:41] (03CR) 10Joal: "One comment and one question :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1213572 (https://phabricator.wikimedia.org/T409584) (owner: 10Mforns) [09:37:56] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28): Create alert on Airflow scheduler slow down - https://phabricator.wikimedia.org/T411405#11422788 (10Gehel) [09:38:18] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Create alert on Airflow scheduler slow down - https://phabricator.wikimedia.org/T411405#11422790 (10Gehel) [11:35:27] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11423167 (10Marostegui) [11:35:31] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11423168 (10Marostegui) [11:36:20] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rev_sha1 from revision table in wmf production - https://phabricator.wikimedia.org/T411164#11423170 (10Marostegui) [11:36:26] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop ar_sha1 from archive table in wmf production - https://phabricator.wikimedia.org/T411163#11423171 (10Marostegui) [12:45:57] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Create new Druid datasource based on the `mediawiki_revision_history_v1` table - https://phabricator.wikimedia.org/T411484 (10JAllemandou) 03NEW [12:46:10] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Create new Druid datasource based on the `mediawiki_revision_history_v1` table - https://phabricator.wikimedia.org/T411484#11423368 (10JAllemandou) a:05amastilovic→03JAllemandou [12:48:14] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work: Create new Druid datasource based on the `mediawiki_revision_history_v1` table - https://phabricator.wikimedia.org/T411484#11423376 (10JAllemandou) [12:49:16] (03CR) 10Mforns: Modify ja3n_ua_hourly queries to migrate the table to Iceberg (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1213572 (https://phabricator.wikimedia.org/T409584) (owner: 10Mforns) [13:00:21] (03PS1) 10Joal: Add compute_mediawiki_revision_history_reduced HQL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1214023 (https://phabricator.wikimedia.org/T411484) [13:05:17] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11423504 (10brouberol) ` brouberol@kafka-main2008:~/T407185$ kafka reassign-partitions kafka reassign-partitions --reassignment-json-file ./... [13:33:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:33:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:38:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:38:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:39:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:39:12] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:39:49] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop modtoken and flags from cache tables - https://phabricator.wikimedia.org/T411497 (10Ladsgroup) 03NEW [13:46:29] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop modtoken and flags from cache tables - https://phabricator.wikimedia.org/T411497#11423692 (10Marostegui) p:05Triage→03Medium a:03Marostegui I remember seeing them in parsercache, I will check anyway. [13:47:00] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop modtoken and flags from cache tables - https://phabricator.wikimedia.org/T411497#11423699 (10Marostegui) ` $ sudo db-mysql pc1011 parsercache -e "show tables like 'objec%'"; +--------------------------------+ | Tables_in_parsercache (objec%) | +-... [13:51:13] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop modtoken and flags from cache tables - https://phabricator.wikimedia.org/T411497#11423716 (10Ladsgroup) >>! In T411497#11423699, @Marostegui wrote: > ` > $ sudo db-mysql pc1011 parsercache -e "show tables like 'objec%'"; > +----------------------... [14:21:30] (03CR) 10Xcollazo: "Wow this SQL is surprisingly simple. Not sure why I thought it will be quite complicated!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1214023 (https://phabricator.wikimedia.org/T411484) (owner: 10Joal) [14:23:46] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11423873 (10brouberol) One thing to note: we're seeing a lot of traffic already coming out of kafka-main2010, and the rebalancing will add n... [14:27:31] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop modtoken and flags from cache tables - https://phabricator.wikimedia.org/T411497#11423885 (10Marostegui) Yeah they aren't written since 2022 from what I can see [14:30:28] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop modtoken and flags from cache tables - https://phabricator.wikimedia.org/T411497#11423891 (10Marostegui) @Ladsgroup I am confused, the patch you linked talks about `objectcache` table but ms sections do not have that, they have `objectstash` inst... [14:34:36] aloha! I'm looking at why seemingly im not getting / seeing events in "event"."mwcli_command_execute" that I was / am expecting :/ [14:34:47] as far as I can see nothing has changed on my end [14:36:06] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop modtoken and flags from cache tables - https://phabricator.wikimedia.org/T411497#11423915 (10Ladsgroup) So the tables are built from the structure of objectcache table. They take the schema of objectcache and use it template to create other table... [14:37:50] Apparnetly I get a 403 D: [14:39:18] Please set a user-agent and respect our robot policy dammit :P [16:13:38] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424460 (10brouberol) {F70821449} Storage has been rebalanced on kafka-main. I'll let traffic settle, and will see whether we need to adjus... [16:33:59] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424574 (10brouberol) I've switched leadership for the `codfw.mediawiki.job.htmlCacheUpdate` topic. ` brouberol@kafka-main2008:~/T407185$... [16:34:06] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424576 (10brouberol) 05In progress→03Resolved [16:35:39] 06Data-Engineering, 06Test Kitchen: json schema tools: Can we allow changing the regex pattern for a schema field - https://phabricator.wikimedia.org/T411518 (10Milimetric) 03NEW [16:39:08] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424636 (10Clement_Goubert) Thanks for all the support @brouberol <3 [16:46:53] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424674 (10brouberol) Anytime! [16:58:35] (03PS2) 10Mforns: Modify ja3n_ua_hourly queries to migrate the table to Iceberg [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1213572 (https://phabricator.wikimedia.org/T409584) [16:59:09] (03CR) 10Mforns: Modify ja3n_ua_hourly queries to migrate the table to Iceberg (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1213572 (https://phabricator.wikimedia.org/T409584) (owner: 10Mforns) [17:11:32] (03CR) 10Xcollazo: [C:03+1] Modify ja3n_ua_hourly queries to migrate the table to Iceberg [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1213572 (https://phabricator.wikimedia.org/T409584) (owner: 10Mforns) [17:12:38] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop modtoken and flags from cache tables - https://phabricator.wikimedia.org/T411497#11424934 (10Krinkle) On ParserCache hosts we instantiate the objectcache template 256 times as pcXXX, like `pc128` ([wmf-config](https://gerrit.wikimedia.org/g/opera... [17:43:05] 14Analytics, 06Data-Engineering, 06Test Kitchen: Count the number of video plays - https://phabricator.wikimedia.org/T198628#11425163 (10Milimetric) we're happy to consult as this work progresses, not ignoring it :) [17:47:39] 06Data-Engineering, 06Test Kitchen, 10Event-Platform: Document instructions for deleting an event stream and its usages - https://phabricator.wikimedia.org/T360210#11425179 (10Milimetric) let me know if we need to actively do anything here [17:52:29] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 10MediaWiki-extensions-WikimediaEvents, and 3 others: Decide on how data platform wants to monitor bundle sizes - https://phabricator.wikimedia.org/T378772#11425270 (10Milimetric) I don't think Experiment Platform has owners... [17:53:42] 14Analytics, 06Data-Engineering, 06Test Kitchen: Count the number of video plays - https://phabricator.wikimedia.org/T198628#11425277 (10Doc_James) Okay so we have video and audio views live and functional on Basque Wikipedia. https://commons-play-listener.toolforge.org/dashboard/?website=eu.wikipedia.org... [18:07:39] (03CR) 10Mforns: [V:03+2 C:03+2] "Merging." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1213572 (https://phabricator.wikimedia.org/T409584) (owner: 10Mforns) [18:08:14] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Set a custom From: email address for alerts from Airflow dev instances - https://phabricator.wikimedia.org/T411536 (10amastilovic) 03NEW [18:21:05] (03PS2) 10Joal: Add compute_mediawiki_revision_history_reduced HQL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1214023 (https://phabricator.wikimedia.org/T411484) [18:23:00] (03CR) 10Joal: "Thanks for the review @xcollazo@wikimedia.org. Can you please review the comment? I think we're good after that." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1214023 (https://phabricator.wikimedia.org/T411484) (owner: 10Joal) [18:33:31] (03CR) 10Xcollazo: [C:03+1] Add compute_mediawiki_revision_history_reduced HQL (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1214023 (https://phabricator.wikimedia.org/T411484) (owner: 10Joal) [18:34:54] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Review and productionize the WME differential privacy data set - https://phabricator.wikimedia.org/T409601#11425452 (10Snwachukwu) @Htriedman, Regarding the request to move the code to a non-user repo, I'm a little confused, It seems it has already bee... [19:15:06] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work, 13Patch-For-Review: Provide a Spark production access for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11425647 (10BTullis) I'm still making steady progress on this, but t... [19:31:25] 06Data-Engineering, 10Wmfdata-Python, 06Movement-Insights (FY25-26 H1): Update Wmfdata developer documentation to correctly cover working with UV - https://phabricator.wikimedia.org/T402990#11425740 (10nshahquinn-wmf) 05Open→03Resolved [21:06:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:06:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:39:44] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10MediaWiki-Page-derived-data, 07OKR-Work, 13Patch-For-Review: Create new Druid datasource based on the `mediawiki_revision_history_v1` table - https://phabricator.wikimedia.org/T411484#11426141 (10xcollazo) I used ` joal.test_mediawiki_revision_... [22:31:07] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Review and productionize the WME differential privacy data set - https://phabricator.wikimedia.org/T409601#11426307 (10Htriedman) Hi @Snwachukwu! I'll try to answer your questions one-by-one: > Can you help me confirm that these are all the list of ou...