[03:03:52] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#12043501 (10Ahoelzl) [03:18:36] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Add Cassandra loading capability in dbt dags - https://phabricator.wikimedia.org/T429862 (10AKhatun_WMF) 03NEW [03:27:10] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#12043519 (10AKhatun_WMF) [03:27:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Add Cassandra loading capability in dbt dags - https://phabricator.wikimedia.org/T429862#12043518 (10AKhatun_WMF) [03:33:30] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Document editor counts table and APIs - https://phabricator.wikimedia.org/T429863 (10AKhatun_WMF) 03NEW [03:42:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Document editor counts table and APIs - https://phabricator.wikimedia.org/T429863#12043540 (10AKhatun_WMF) [[https://wikimedia.slack.com/archives/C05RHK7PS6Q/p1781795528388069 | Slack thread]] From @Ottomata > I think you can make a wikitech page, under Da... [08:12:20] (03PS1) 10Seanleong-wmde: Remove the type [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1305037 (https://phabricator.wikimedia.org/T426384) [08:23:21] (03PS1) 10Seanleong-wmde: Remove the type [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1305039 (https://phabricator.wikimedia.org/T426384) [09:04:37] (03CR) 10Awight: [C:03+2] "Yes. The machines are using a version of PHP < 8.0 where the `mixed` type is introduced so this patch is necessary." [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1305037 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [09:05:20] (03Merged) 10jenkins-bot: Remove the type [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1305037 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [09:17:56] (03CR) 10Awight: [C:03+2] Remove the type [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1305039 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [09:18:26] (03Merged) 10jenkins-bot: Remove the type [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1305039 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [09:18:43] 06Data-Engineering: WMF Data Engineering Request: Additional Columns to Geoeditors Tables - https://phabricator.wikimedia.org/T428888#12044326 (10catherine.kelsey.wmde) Hi @JAllemandou - thanks for the question and apologies for the delay in the response (I was on holiday last week)! I think just in the daily ta... [09:25:59] 06Data-Engineering: WMF Data Engineering Request: Additional Columns to Geoeditors Tables - https://phabricator.wikimedia.org/T428888#12044375 (10JAllemandou) As @Ahoelzl mentioned in slack it'd be awesome if you could make the change :) I'll surely review it. [09:32:34] 06Data-Engineering: WMF Data Engineering Request: Additional Columns to Geoeditors Tables - https://phabricator.wikimedia.org/T428888#12044415 (10catherine.kelsey.wmde) Sweet! Will add this to our backlog and bring to our team's planning session next week :) [09:33:22] 06Data-Engineering, 10WMDE Analytics: WMF Data Engineering Request: Additional Columns to Geoeditors Tables - https://phabricator.wikimedia.org/T428888#12044423 (10catherine.kelsey.wmde) a:05Ahoelzl→03catherine.kelsey.wmde [10:16:14] 10Data-Engineering-Roadmap, 06Discovery-Search, 10DPE-Mediawiki-Content, 07Epic, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.20 - https://phabricator.wikimedia.org/T376812#12044607 (10dcausse) 05Open→03Resolved [10:22:45] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734#12044622 (10APizzata-WMF) First example of [[ https://superset.wikimedia.org/superset/dashb... [11:12:06] 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10MediaWiki-extensions-WikimediaEvents, and 3 others: Could not hoist data into experiment.subject_id for event - https://phabricator.wikimedia.org/T421152#12044807 (10phuedx) [11:17:29] 06Data-Engineering, 10Event-Platform: [EventGate] Sanitize stream name in all metrics - https://phabricator.wikimedia.org/T429799#12044816 (10phuedx) [11:42:08] 06Data-Engineering, 10Event-Platform: [EventGate] Add configurable UA denylist - https://phabricator.wikimedia.org/T429898 (10phuedx) 03NEW [12:25:14] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045103 (10brouberol) Sorry, I'm only seeing this now! Indeed, I can see the pagetitles and mediatitles DAG having failed for a... [12:25:34] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045105 (10brouberol) a:03brouberol [12:52:18] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045223 (10brouberol) The dump pod is stuck in `ContainerCreating` phase. ` mediawiki-mediatitles-dump-ns6... [12:58:25] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045263 (10brouberol) I noticed that the pod was always scheduled to the same host. I cordoned the host, re-ran the DAG, and the... [13:07:38] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045327 (10brouberol) I'm not 100% sure why the OOM didn't propagate back to the kubectl watch stream when the pod was running o... [13:10:47] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045349 (10brouberol) These resources requests/limits come directly from the base mediawiki job template, and thus need an overr... [13:15:23] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Wikimedia Enterprise: PageViews S3 Data Transfer MR [Enterprise] - https://phabricator.wikimedia.org/T425543#12045397 (10LDlulisa-WMF) @Ahoelzl I am not blocked on this ticket specifically. But, I'd like to do a final test of the entire pipeline on the d... [13:15:28] 06Data-Engineering, 06Data-Persistence, 06Data-Platform-SRE, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282#12045399 (10Gehel) For the medium or long term solution, whether it is improving the current jobs or migrating to dumps b... [13:42:44] 06Data-Engineering, 10Kafka-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Incident Severity 3, 07Wikimedia-Incident: staging.webrequest.page_view.dev0 taking up most space on kafka-jumbo - https://phabricator.wikimedia.org/T429088#12045566 (10MLechvien-WMF) [13:44:27] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045589 (10brouberol) ^ this ^ change has led to the pod being stable in my airflow dev environment. It wi... [13:44:46] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045595 (10brouberol) 05Open→03In progress [13:56:53] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045724 (10brouberol) ` brouberol@deploy1003:~$ kubectl get pod mediawiki-mediatitles-dump-ns6 -o json | j... [14:02:21] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12045750 (10AKhatun_WMF) @RKemper we can go ahead wth your plan of stateless deployment. Let's find... [14:06:54] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#12045769 (10Ottomata) If we do this, we should do it for all wikis, not... [14:13:04] !log Deploying Refinery as part of weekly deployment train [14:13:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:13:07] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045802 (10brouberol) {F90141174} The mediatitles dump completed! https://dumps.wikimedia.org/other/mediat... [14:13:49] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045809 (10brouberol) {F90141301} The 16GB mem limit was a good guess. I might be good to increase it a bit/ [14:25:09] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045876 (10brouberol) The pagetitles dump has completed as well! {F90142313} {F90142343} https://dumps... [14:26:53] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: New mediatitles and pagetitles dumps are missing - https://phabricator.wikimedia.org/T427195#12045884 (10brouberol) 05In progress→03Resolved [14:34:14] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product-Analytics: dbt-jobs backfill: PP3 API hourly and known clients aggregate jobs - https://phabricator.wikimedia.org/T429341#12045932 (10amastilovic) Status update: mrt_api_requests_hourly: [x] 2026-02-01 to to 2026-03-31 (Feb and Mar can be done... [14:56:30] !log Deployed Refinery as part of weekly deployment train [14:56:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:07:15] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#12046100 (10dr0ptp4kt) >>! In T426347#12045768, @Ottomata wrote: > If we... [15:22:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 07Epic: Ingest wmf_mediawiki tables to datahub - https://phabricator.wikimedia.org/T429931 (10Ottomata) 03NEW [15:31:53] 06Data-Engineering, 10observability, 10Observability-Metrics, 10Event-Platform: Enable querying operational (prometheus) metrics via the WMF Data Platform - https://phabricator.wikimedia.org/T390328#12046281 (10Milimetric) +1 to needing this - I'm in a rush right now trying to pull a few Prometheus metrics... [15:35:53] (03PS1) 10Gerrit maintenance bot: Add isv.wikipedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1305156 (https://phabricator.wikimedia.org/T429935) [15:36:24] (03PS1) 10Gerrit maintenance bot: Add min.wikiquote to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1305158 (https://phabricator.wikimedia.org/T429943) [15:37:13] (03PS1) 10Gerrit maintenance bot: Add bol.wikipedia to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1305162 (https://phabricator.wikimedia.org/T429951) [15:56:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 07Essential-Work, 10Event-Platform: jsonschema-tools should not consider definitions fields in compatibility checks. - https://phabricator.wikimedia.org/T425028#12046635 (10Ottomata) a:05Ottomata→03tchin [16:35:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 07Epic, 13Patch-For-Review: Ingest wmf_mediawiki tables to datahub - https://phabricator.wikimedia.org/T429931#12046992 (10Ottomata) a:03Ottomata [16:58:51] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for later deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1305158 (https://phabricator.wikimedia.org/T429943) (owner: 10Gerrit maintenance bot) [16:59:39] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for later deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1305162 (https://phabricator.wikimedia.org/T429951) (owner: 10Gerrit maintenance bot) [17:00:34] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for later deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1305156 (https://phabricator.wikimedia.org/T429935) (owner: 10Gerrit maintenance bot) [17:07:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Monitor webrequest data loss on a dashboard - https://phabricator.wikimedia.org/T429972 (10Ahoelzl) 03NEW [17:07:50] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Monitor webrequest data loss on a dashboard - https://phabricator.wikimedia.org/T429972#12047254 (10Ahoelzl) [17:07:55] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#12047255 (10Ahoelzl) [17:08:50] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#12047298 (10Ahoelzl) At this point lets increase the alerting thresholds to reduce the noise on the data engineering alert mailing list, indiv... [17:09:04] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#12047299 (10Ahoelzl) a:05Ahoelzl→03JAllemandou [17:12:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Wikidata, 10WMDE Analytics: WMF Data Engineering Request: Additional Columns to Geoeditors Tables - https://phabricator.wikimedia.org/T428888#12047317 (10Ahoelzl) [17:53:22] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#12047549 (10Ahoelzl) a:05dr0ptp4kt→03Snwachukwu [18:13:22] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#12047624 (10Ahoelzl) a:05JAllemandou→03Snwachukwu [18:33:18] 06Data-Engineering, 06Traffic: Tune refine webrequest data loss threshold to avoid noisy irrelevant alerts. - https://phabricator.wikimedia.org/T429809#12047684 (10Ahoelzl) a:03Snwachukwu [18:33:28] 06Data-Engineering, 06Java-Scala-Standardization, 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#12047687 (10TheDJ) Can this be closed ? [19:37:20] 06Data-Engineering, 10Event-Platform: Flink - expose Kafka consumer metrics - https://phabricator.wikimedia.org/T429994 (10Ottomata) 03NEW [19:37:50] 06Data-Engineering, 10Event-Platform: Flink - expose Kafka consumer metrics - https://phabricator.wikimedia.org/T429994#12047927 (10Ottomata) [19:37:51] 06Data-Engineering, 10Event-Platform: Flink - expose Kafka consumer metrics - https://phabricator.wikimedia.org/T429994#12047928 (10Ottomata) p:05Triage→03Low [19:39:51] 06Data-Engineering, 10Test Kitchen, 07Essential-Work: Improve instrument event data data lake management - https://phabricator.wikimedia.org/T429385#12047933 (10mpopov) > So overall the story for experiment owners becomes: > > 1. Create your experiment. Instrument :) Experiment data is already going into `w... [19:42:52] 06Data-Engineering, 10Test Kitchen, 07Essential-Work: Improve instrument event data data lake management - https://phabricator.wikimedia.org/T429385#12047953 (10mpopov) @GGoncalves-WMF: By the way, @SNowick_WMF would benefit from this solution greatly because she's been doing analysis of the mobile app Centr... [20:40:05] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Make cusi_user and cusi_case table information available in the data lake - https://phabricator.wikimedia.org/T429703#12048130 (10Ahoelzl) a:05Ahoelzl→03cchen [20:40:27] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Make cusi_user and cusi_case table information available in the data lake - https://phabricator.wikimedia.org/T429703#12048132 (10Ahoelzl) @cchen are you unblocked with the event stream approach? [20:41:33] 06Data-Engineering: dbt-jobs backfill: all base models for moderator actions - https://phabricator.wikimedia.org/T429995 (10CMyrick-WMF) 03NEW [20:42:30] 06Data-Engineering: dbt-jobs backfill: all base models for moderator actions - https://phabricator.wikimedia.org/T429995#12048158 (10CMyrick-WMF) [21:01:15] 06Data-Engineering: dbt-jobs backfill: all base models for moderator actions - https://phabricator.wikimedia.org/T429995#12048219 (10CMyrick-WMF) [21:04:02] 06Data-Engineering: Operationalize dbt models: automate monthly updates for moderator metrics - https://phabricator.wikimedia.org/T429997 (10CMyrick-WMF) 03NEW [21:04:54] 06Data-Engineering: Operationalize dbt models for monthly Moderators and UWERs metrics - https://phabricator.wikimedia.org/T429997#12048249 (10CMyrick-WMF) [21:06:15] FIRING: HdfsRpcQueueLength: RPC queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength [21:21:15] RESOLVED: HdfsRpcQueueLength: RPC queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength [23:55:59] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list June 2026 - https://phabricator.wikimedia.org/T430008 (10GFontenelle_WMF) 03NEW