[01:53:05] 06Data-Engineering, 10Data-Platform, 06Data-Platform-SRE, 07Epic: [Epic] Migrate the Data Lake from Hadoop to Kubernetes and Ceph/S3 - https://phabricator.wikimedia.org/T435461#12240013 (10BTullis) a:03BTullis [08:25:19] 06Data-Engineering, 06Data-Platform-SRE: Move the remaining HDFS consumers (Druid deep storage, dumps distribution) away from HDFS - https://phabricator.wikimedia.org/T435496#12240419 (10BTullis) [08:25:20] 06Data-Engineering, 06Data-Platform-SRE, 07Epic: Migrate the Data Lake datasets from HDFS to Ceph/S3 - https://phabricator.wikimedia.org/T435510#12240418 (10BTullis) [08:25:21] 06Data-Engineering, 06Data-Platform-SRE: Data Lake ingestion after Hadoop: decide the future of Sqoop and Gobblin - https://phabricator.wikimedia.org/T435489#12240420 (10BTullis) [08:29:49] 06Data-Engineering, 06Data-Platform-SRE: Enable OIDC authentication and IAM authorization on the Data Platform Ceph clusters - https://phabricator.wikimedia.org/T435470#12240434 (10BTullis) [08:30:27] 06Data-Engineering, 10Data-Platform, 06Data-Platform-SRE: Implement group-level authorization in the query engines (Spark and Presto) - https://phabricator.wikimedia.org/T435475#12240436 (10BTullis) [08:31:44] 06Data-Engineering, 06Data-Platform-SRE, 07Epic: Migrate the Data Lake datasets from HDFS to Ceph/S3 - https://phabricator.wikimedia.org/T435510#12240439 (10BTullis) [08:31:45] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 07Essential-Work: Do performance testing of a big Hadoop Table hosted by Ceph - https://phabricator.wikimedia.org/T381416#12240438 (10BTullis) [08:31:46] 06Data-Engineering, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Enable Ceph S3 locations for Hive Metastore tables - https://phabricator.wikimedia.org/T425673#12240440 (10BTullis) [08:56:22] (03CR) 10A-pizzata: refinery-job: Add HiveToJdbc job to mirror Hive tables to PostgreSQL (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1327582 (https://phabricator.wikimedia.org/T432017) (owner: 10A-pizzata) [09:28:07] (03PS2) 10A-pizzata: refinery-job: Add HiveToJdbc job to mirror Hive tables to PostgreSQL [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1327582 (https://phabricator.wikimedia.org/T432017) [09:28:40] (03CR) 10A-pizzata: refinery-job: Add HiveToJdbc job to mirror Hive tables to PostgreSQL (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1327582 (https://phabricator.wikimedia.org/T432017) (owner: 10A-pizzata) [09:52:53] (03CR) 10Milimetric: [C:03+2] refinery-job: Add HiveToJdbc job to mirror Hive tables to PostgreSQL [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1327582 (https://phabricator.wikimedia.org/T432017) (owner: 10A-pizzata) [09:53:36] (03CR) 10A-pizzata: [V:03+2] refinery-job: Add HiveToJdbc job to mirror Hive tables to PostgreSQL [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1327582 (https://phabricator.wikimedia.org/T432017) (owner: 10A-pizzata) [09:58:29] 06Data-Engineering, 06Data-Platform-SRE: Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590 (10BTullis) 03NEW [10:01:24] 06Data-Engineering, 06Data-Platform-SRE: Allow unauthenticated access to the dse-k8s OIDC discovery and JWKS endpoints - https://phabricator.wikimedia.org/T435591 (10BTullis) 03NEW [10:05:40] 06Data-Engineering, 06Data-Platform-SRE: Validate the full token-for-keys authentication flow end to end in codfw - https://phabricator.wikimedia.org/T435593 (10BTullis) 03NEW [10:08:55] (03Merged) 10jenkins-bot: refinery-job: Add HiveToJdbc job to mirror Hive tables to PostgreSQL [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1327582 (https://phabricator.wikimedia.org/T432017) (owner: 10A-pizzata) [10:25:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: data lake: git commit history - https://phabricator.wikimedia.org/T433696#12240749 (10Antoine_Quhen) OK * commits is now an optional list * interaction_type allows multiple kind of interactions [10:37:08] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review, 10Event-Platform, 13Patch-For-Review: [Data Persistence Design Review] Relative trending pages - https://phabricator.wikimedia.org/T432352#12240767 (10JMonton-WMF) #### Using Object Cache So, we are changing the a... [11:31:41] 06Data-Engineering, 06Data-Platform-SRE: [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604 (10BTullis) 03NEW [11:51:35] 06Data-Engineering, 06Data-Platform-SRE: Decide upon and deploy the table catalog for the S3 data lake - https://phabricator.wikimedia.org/T435474#12241033 (10BTullis) >>! In T435474#12237194, @Ottomata wrote: > A tech evaluation of metastore options may help us make this decision. Yes, that is a good idea. I... [11:56:49] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12241063 (10Marostegui) [11:56:56] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12241065 (10Marostegui) 05Open→03Resolved a:03Marostegui This is done [11:58:44] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12241074 (10Marostegui) [12:06:03] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop two columns from transcode table - https://phabricator.wikimedia.org/T435064#12241103 (10Marostegui) [12:12:43] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07Epic: Move production spark jobs from YARN to k8s - https://phabricator.wikimedia.org/T432322#12241161 (10BTullis) [12:13:10] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE, 07OKR-Work, 13Patch-For-Review: Provide a Spark-on-k8s access for sql tools (dbt) - https://phabricator.wikimedia.org/T410017#12241163 (10BTullis) [12:13:40] 06Data-Engineering, 06Data-Platform-SRE, 07Epic: Migrate the Data Lake datasets from HDFS to Ceph/S3 - https://phabricator.wikimedia.org/T435510#12241165 (10BTullis) p:05Triage→03High [12:42:27] 06Data-Engineering, 06Data-Engineering-Icebox, 06Connection-Team, 06Movement-Insights, 05Campaign-Registration: Develop a consistent rule for which special pages count as pageviews - https://phabricator.wikimedia.org/T240676#12241218 (10pmiazga) [12:55:44] 06Data-Engineering, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12241251 (10BTullis) [13:05:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027: Move all FY2026-2027 chart data to PostgreSQL - https://phabricator.wikimedia.org/T432017#12241283 (10APizzata-WMF) [13:19:09] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review, 10Event-Platform, 13Patch-For-Review: [Data Persistence Design Review] Relative trending pages - https://phabricator.wikimedia.org/T432352#12241303 (10Ladsgroup) Do you have an estimate of number of writes or reads... [13:51:59] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review, 10Event-Platform, 13Patch-For-Review: [Data Persistence Design Review] Relative trending pages - https://phabricator.wikimedia.org/T432352#12241377 (10JMonton-WMF) @Ottomata added some numbers here: https://phabric... [14:30:26] 06Data-Engineering, 10Test Kitchen, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#12241479 (10mpopov) [14:42:09] 06Data-Engineering, 13Patch-For-Review: Enable Refine event tables to Iceberg - https://phabricator.wikimedia.org/T417176#12241604 (10Antoine_Quhen) [[ https://docs.google.com/document/d/1rEGUK_zeNelIkFBfLIdV-t9pM-PeP-Am5frku8Bgvgk/edit?tab=t.0#heading=h.99wcurwxv0pw | Here is a decision brief to discuss the t... [15:28:23] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review, 10Event-Platform, 13Patch-For-Review: [Data Persistence Design Review] Relative trending pages - https://phabricator.wikimedia.org/T432352#12241834 (10JMonton-WMF) Abut **Read rate**: From this Slack thread: https... [17:17:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): MWH `page_is_deleted` and `revision_is_deleted_py_page_deletion` are not consistent - https://phabricator.wikimedia.org/T430543#12242209 (10xcollazo) Great investigation @AKhatun_WMF ! WDYT we should do? Should we document the behavior? Should we attemp... [17:28:50] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Create new mediawiki.tags_change event stream - https://phabricator.wikimedia.org/T434143#12242259 (10tchin) Hmm even though we have the log_id already, there doesn't seem to be a value class for a log entry that we're able to use? Th... [17:41:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Add source='snapshot+events' for incrementally-enriched snapshot rows - https://phabricator.wikimedia.org/T428969#12242345 (10xcollazo) 05Open→03Declined Considering that the proposed work in this ticket wou... [17:48:29] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 07Epic: Incremental MediaWiki History - v2 - https://phabricator.wikimedia.org/T432025#12242392 (10xcollazo) Another problem that we should think about for v2 is the issue from {T428969}: How can a consumer efficiently update if all they want to consu... [19:07:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:12:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12242532 (10AKhatun_WMF) [19:18:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12242547 (10AKhatun_WMF) [19:18:35] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12242548 (10AKhatun_WMF) Average hourly citation and reference count changes **across wikipedias**: ` spark.sql(""" SELECT AVG(cite_inserted), AVG(... [19:22:50] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12242564 (10AKhatun_WMF) [19:32:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:56:43] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 07OKR-Work (WE1 FY2025-26): [Spike] Adding access_method metadata to moderator action event streams - https://phabricator.wikimedia.org/T419019#12242660 (10mpopov) I really like the proposed `server_interface` and `client_software` with the listed poss... [21:02:36] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Backfills, Delayed runs, and Quality Checks - https://phabricator.wikimedia.org/T433110#12242792 (10Ahoelzl) [21:04:23] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#12242795 (10Ahoelzl) a:05Ottomata→03tchin [21:05:07] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:19:14] 06Data-Engineering, 06Data-Engineering-Radar, 10Wmfdata-Python, 07Epic: Enable Wmfdata-Python to access MariaDB replicas from the cluster - https://phabricator.wikimedia.org/T340467#12242844 (10mpopov) 05Open→03Invalid [21:19:16] 06Data-Engineering, 06Data-Engineering-Radar, 10Wmfdata-Python: Let user specify cnf to use when connecting to MariaDB - https://phabricator.wikimedia.org/T340469#12242846 (10mpopov) 05Open→03Invalid [21:19:43] 06Data-Engineering, 06Data-Engineering-Radar, 10Wmfdata-Python: Retrieve host & port info when connecting to MariaDB replicas on the cluster - https://phabricator.wikimedia.org/T340472#12242850 (10mpopov) 05Open→03Invalid [21:20:04] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Product Analytics ETL Migration: Pilot (MediaSearch ETLs) - https://phabricator.wikimedia.org/T333208#12242855 (10mpopov) 05Open→03Invalid [21:20:11] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 07Epic: Notebook Scheduler for Product Analytics - https://phabricator.wikimedia.org/T322532#12242857 (10mpopov) 05Open→03Invalid [21:30:07] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [22:01:07] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 06Product-Analytics, 10superset.wikimedia.org: Improve Superset's error message for draft dashboards - https://phabricator.wikimedia.org/T405535#12242958 (10mpopov)