[00:16:30] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Dbt backfill: editor_month and base_account_registration - https://phabricator.wikimedia.org/T430602#12078534 (10nshahquinn-wmf) @amastilovic I tested `wmf_contributors.base_account_registration` and it looks good! However, `wmf_contributors.editor_month` s... [01:48:51] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Privacy Engineering, and 2 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12078674 (10aranyap) The Product Safety & Integrity team has reviewed this table and is considering this to be a low privacy ris... [02:04:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.10; 2026-07-07), 13Patch-For-Review: mediawiki.page_change.v1 event - Add revision revert details - https://phabricator.wikimedia.org/T423583#12078689 (10Ottomata) @tchin T423583#11914122 I forgot I want... [02:09:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 06Machine-Learning-Team (Q4 FY2025-26): `mediawiki.page_change.v1`: negative namespace_id schema validation errors - https://phabricator.wikimedia.org/T421237#12078708 (10Ottomata) [02:38:22] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Privacy Engineering, and 2 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12078728 (10Soda) >>! In T344108#12078673, @aranyap wrote: > The Product Safety & Integrity team has reviewed this table and is... [04:11:31] 14Data-Engineering (Q3 2025 January 1st - March 31th), 10service-utils: Fix service-utils metrics routing naming discrepancy - https://phabricator.wikimedia.org/T387824#12078789 (10tchin) [04:16:17] 14Data-Engineering (Q2 2024 October 1st - December 31th), 10Observability-Tracing, 10service-utils: service-utils helper for trace header propagation - https://phabricator.wikimedia.org/T371120#12078790 (10tchin) [04:16:22] 14Data-Engineering (Q3 2025 January 1st - March 31th), 10service-utils: Replace service runner with a simplified library to better support metrics and debugging: service-utils - https://phabricator.wikimedia.org/T360924#12078791 (10tchin) [04:17:17] 14Data-Engineering (Q3 2025 January 1st - March 31th), 10service-utils: Update eventstreams Grafana Dashboards to use histogram for router metrics - https://phabricator.wikimedia.org/T386204#12078792 (10tchin) [05:54:15] FIRING: HdfsRpcQueueLength: RPC queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength [05:59:15] RESOLVED: HdfsRpcQueueLength: RPC queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength [07:31:27] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734#12079046 (10APizzata-WMF) [07:31:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 07Epic, 13Patch-For-Review: Incremental MediaWiki History Phase I - https://phabricator.wikimedia.org/T424350#12079048 (10APizzata-WMF) [09:01:32] (03CR) 10Joal: "3 nits to discuss" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1306491 (https://phabricator.wikimedia.org/T430020) (owner: 10KCVelaga) [09:12:36] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#12079528 (10APizzata-WMF) In this month's run there were some hiccups (more in ). Anyway I... [09:15:31] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Enable `upsert-kafka` tables in `eventutilities-flink - https://phabricator.wikimedia.org/T430919 (10JMonton-WMF) 03NEW [09:23:55] (03CR) 10KCVelaga: DDL for wmf_traffic.user_agents_info table. (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1306491 (https://phabricator.wikimedia.org/T430020) (owner: 10KCVelaga) [10:32:13] o/ I'm doing a little cleanup on old nrpe checks and I noticed that there's still a bit of turnilo stuff in puppet. Can that all be removed wholesale? [12:17:54] 06Data-Engineering: sanitization re-run request: event_sanitized.mediawiki_page_html_feature_counts_change_v1 - https://phabricator.wikimedia.org/T430752#12080398 (10CMyrick-WMF) 05Open→03Resolved I just checked, and the row counts for May 16 and May 17 match now. 🎉 [12:32:32] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): dbt-jobs backfill: all base models for moderator actions - https://phabricator.wikimedia.org/T429995#12080510 (10CMyrick-WMF) Per the updated metadata noted in the table above, These models need backfilling for 2020-07-01 through 2025-11-01 (inclusive) - b... [12:53:26] (03CR) 10Joal: DDL for wmf_traffic.user_agents_info table. (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1306491 (https://phabricator.wikimedia.org/T430020) (owner: 10KCVelaga) [13:13:55] (03PS2) 10KCVelaga: DDL for wmf_traffic.user_agents_info table. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1306491 (https://phabricator.wikimedia.org/T430020) [13:16:06] (03CR) 10KCVelaga: "Updated the DDL." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1306491 (https://phabricator.wikimedia.org/T430020) (owner: 10KCVelaga) [13:18:15] FIRING: HdfsRpcQueueLatency: RPC queue latency on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=56&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLatency [13:26:53] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: mediawiki.page_change.v1 event - Add user first_registration_dt field - https://phabricator.wikimedia.org/T426998#12080733 (10Ottomata) > If you'd like to get the "globally first registration" da... [13:28:15] RESOLVED: HdfsRpcQueueLatency: RPC queue latency on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=56&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLatency [13:55:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: MediaWiki state change events - emit global edit count - https://phabricator.wikimedia.org/T430967 (10Ottomata) 03NEW [14:07:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic: Tune refine webrequest data loss threshold to avoid noisy irrelevant alerts. - https://phabricator.wikimedia.org/T429809#12081034 (10Snwachukwu) I made the following changes to the threshold: **data_loss_warning** - data_loss_warning: prev... [14:59:51] 06Data-Engineering, 06Data-Persistence, 06Data-Platform-SRE, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282#12081372 (10Gehel) Ping us if more support from DPE SRE is needed. Otherwise, I'm expecting Data Engineering (@Ahoelzl) t... [15:10:15] 06Data-Engineering, 10Test Kitchen, 07Essential-Work, 10Event-Platform: [EventGate] Add configurable UA denylist - https://phabricator.wikimedia.org/T429898#12081460 (10KReid-WMF) [15:12:36] !log Test Kitchen experiment (poll 112081) - adds: logged-out-retention-round19; removes: none; fields: none - TK tips at https://w.wiki/_cvdP [15:12:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:19:32] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 07Essential-Work: Inconsistent wiki list: grouped_wikis.csv extended *after* some sqoop jobs have already started - https://phabricator.wikimedia.org/T425385#12081533 (10Snwachukwu) Right now this is what the query that updates that table looks like: ` C... [16:13:36] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 07Essential-Work: Inconsistent wiki list: grouped_wikis.csv extended *after* some sqoop jobs have already started - https://phabricator.wikimedia.org/T425385#12081878 (10dr0ptp4kt) Looping @JMonton-WMF who is now the Ops Week on-rotation person. [17:05:49] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.10; 2026-07-07): mediawiki.page_change.v1 event - Add revision revert details - https://phabricator.wikimedia.org/T423583#12082214 (10Ottomata) Tested and works in beta! Did some manual edits and a manual rev... [18:41:46] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 07Essential-Work: Inconsistent wiki list: grouped_wikis.csv extended *after* some sqoop jobs have already started - https://phabricator.wikimedia.org/T425385#12082664 (10Snwachukwu) I have updated the Ops week page to reflect the current state of sqoop lis... [22:46:19] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Right-size Spark resource config using History Server data - https://phabricator.wikimedia.org/T428966#12083708 (10Ahoelzl) 05Open→03Resolved [22:46:22] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic: Tune refine webrequest data loss threshold to avoid noisy irrelevant alerts. - https://phabricator.wikimedia.org/T429809#12083707 (10Ahoelzl) 05Open→03Resolved [22:46:36] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Add Wikitech documentation for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T428824#12083713 (10Ahoelzl) 05Open→03Resolved [22:49:49] 06Data-Engineering, 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12083721 (10bd808)