[05:53:19] 06Data-Engineering, 06Data-Persistence, 06Data-Platform-SRE, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282#12032327 (10Marostegui) @BTullis es6 one yes, I am currently now doing es7 which I guess will be hit by this too once it... [05:55:37] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10MediaWiki-REST-API, 10Tool-wikimedia-attribution, 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)): Check Editor Counts - https://phabricator.wikimedia.org/T427548#12032328 (10KineticPelagic) [07:30:03] (03CR) 10Joal: "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1302824 (https://phabricator.wikimedia.org/T428928) (owner: 10Joal) [07:50:01] 06Data-Engineering, 06Discovery-Search, 10Event-Platform: Upgrade flink-utilities to flink 2.0.2 - https://phabricator.wikimedia.org/T429565 (10dcausse) 03NEW [07:56:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics: [Infra] Sending Apache Spark metrics to PushGateway - https://phabricator.wikimedia.org/T297231#12032468 (10JAllemandou) [07:57:08] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product-Analytics: dbt-jobs backfill: PP3 API hourly and known clients aggregate jobs - https://phabricator.wikimedia.org/T429341#12032470 (10KCVelaga_WMF) [07:59:56] 06Data-Engineering, 06Discovery-Search, 10Event-Platform: Extract eventutilities-flink outside of wikimedia-eventutilities - https://phabricator.wikimedia.org/T429566 (10dcausse) 03NEW [08:11:13] 06Data-Engineering, 06Discovery-Search (2026.06.01 - 2026.07.03), 10Event-Platform: Extract eventutilities-flink outside of wikimedia-eventutilities - https://phabricator.wikimedia.org/T429566#12032528 (10dcausse) a:03dcausse [08:16:01] 06Data-Engineering: event_sanitized.serversideaccountcreation reports users that actually don't exist - https://phabricator.wikimedia.org/T429288#12032544 (10Urbanecm_WMF) >>! In T429288#12030791, @Ottomata wrote: > I'm not sure, but is this a duplicate of {T429061}? In that, it turned out the comparison was jus... [08:20:26] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 07Epic: Bug Fix: generate page-create events and make revisions event type `create` only - https://phabricator.wikimedia.org/T429570 (10JAllemandou) 03NEW [08:21:37] (03CR) 10A-pizzata: [C:03+2] Update Incremental MWH with a fix [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1302824 (https://phabricator.wikimedia.org/T428928) (owner: 10Joal) [08:36:34] (03Merged) 10jenkins-bot: Update Incremental MWH with a fix [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1302824 (https://phabricator.wikimedia.org/T428928) (owner: 10Joal) [08:59:08] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: fix CI error for mediawiki-content-pipelines - https://phabricator.wikimedia.org/T429574 (10APizzata-WMF) 03NEW [09:01:40] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: fix CI error for mediawiki-content-pipelines - https://phabricator.wikimedia.org/T429574#12032733 (10APizzata-WMF) After a [[ https://wikimedia.slack.com/archives/C05RHK7PS6Q/p1781696465965129 | discussion ]] on slack, it was decid... [09:04:43] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: fix CI error for mediawiki-content-pipelines - https://phabricator.wikimedia.org/T429574#12032738 (10APizzata-WMF) [10:01:09] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12032963 (10ops-monitoring-bot) Deployed hiddenparma to alert[1002,2002].wikimedia.org with reason: Change provenance var context... [10:03:40] 06Data-Engineering, 10observability, 06serviceops-radar, 06SRE, and 3 others: Upgrade Kafka to from 1.x to later version - https://phabricator.wikimedia.org/T300102#12032970 (10BTullis) 05Open→03Resolved [10:08:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12032974 (10Fabfur) [10:30:37] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12033025 (10Volans) Do you need anything added to the [[ https://wikitech.wikimedia.org/wiki/Logs/Runbook#Superset_dashboards | su... [10:44:40] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Persistence, 10DiscussionTools: Adapt sqoop configs to account for discussiontools tables only present on some wiki databases - https://phabricator.wikimedia.org/T428916#12033095 (10Marostegui) [10:55:12] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12033148 (10Fabfur) [11:06:12] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Relative Trending - Milestone 3 - Stream & Schema - https://phabricator.wikimedia.org/T429588 (10JMonton-WMF) 03NEW [11:26:50] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Presto cluster improvements for concurrency and workload - https://phabricator.wikimedia.org/T424112#12033250 (10BTullis) a:05amastilovic→03BTullis We had a go at testing this on the... [12:04:08] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: fix CI error for mediawiki-content-pipelines - https://phabricator.wikimedia.org/T429574#12033333 (10APizzata-WMF) Bookworm by itself was not a solution. Had to change also the CI script to use the `OPENJDK_PACKAGE` variable and f... [12:14:15] FIRING: HdfsRpcQueueLength: RPC queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength [12:19:15] RESOLVED: HdfsRpcQueueLength: RPC queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength [12:39:39] (03PS1) 10Joal: Update changelog.md for v0.3.19 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1304040 [12:41:31] (03CR) 10A-pizzata: [C:03+2] Update changelog.md for v0.3.19 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1304040 (owner: 10Joal) [12:51:13] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12033490 (10Fabfur) [12:56:08] (03Merged) 10jenkins-bot: Update changelog.md for v0.3.19 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1304040 (owner: 10Joal) [12:59:37] Starting build #65 for job analytics-refinery-maven-release [13:18:29] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12033588 (10Fabfur) [13:18:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12033590 (10Fabfur) >>! In T427068#12033025, @Volans wrote: > Do you need anything added to the [[ https://wikitech.wikimedia.org/... [13:25:09] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: page_change.v1 increase partitions to 3 - https://phabricator.wikimedia.org/T422511#12033620 (10JMonton-WMF) Through the task: https://phabricator.wikimedia.org/T429127 we stablished a process to decide partitions based on size. This topi... [13:25:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: page_change.v1 increase partitions to 3 - https://phabricator.wikimedia.org/T422511#12033622 (10JMonton-WMF) 05Open→03Declined [13:25:55] Project analytics-refinery-maven-release build #65: 09SUCCESS in 26 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/65/ [13:28:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12033635 (10Fabfur) Checking with @Ahoelzl and @JAllemandou on where to go from there [14:00:31] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Unbalanced partitions in eqiad.mediawiki.content_history_reconcile.v1 topic - https://phabricator.wikimedia.org/T420359#12033883 (10JMonton-WMF) As of today, the data is well balanced: {F89311699} Maybe the unbalanced data was due to the... [14:00:46] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Unbalanced partitions in eqiad.mediawiki.content_history_reconcile.v1 topic - https://phabricator.wikimedia.org/T420359#12033886 (10JMonton-WMF) 05Open→03Invalid [14:05:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12033938 (10APizzata-WMF) the [[ https://airflow.wikimedia.org/dags/mw_content_reconcile_mw_content_history_monthly/grid?la... [14:06:54] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12033942 (10ssingh) Thanks for the work on this @Fabfur! Andreas, is there anything else that needs to be done at our end other than adding this to webr... [14:10:24] !log Test Kitchen experiment (poll 26546) - adds: logged-out-retention-test-growthbook-ncs; removes: none; fields: none - TK tips at https://w.wiki/FwuD [14:10:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:10:44] !log Test Kitchen experiment (poll 26548) - adds: logged-in-retention-test-growthbook; removes: none; fields: none - TK tips at https://w.wiki/FwuD [14:10:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:22:28] !log Test Kitchen experiment (poll 26618) - adds: everyone-cache-splitting-aaa; removes: none; fields: none - TK tips at https://w.wiki/FwuD [14:22:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:27:40] !log Test Kitchen experiment (poll 26649) - adds: none; removes: everyone-cache-splitting-aaa; fields: none - TK tips at https://w.wiki/FwuD [14:27:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:28:10] !log Test Kitchen experiment (poll 26652) - adds: logged-out-retention-test-growthbook-cs; removes: none; fields: none - TK tips at https://w.wiki/FwuD [14:28:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:30:01] !log Test Kitchen experiment (poll 26663) - adds: none; removes: account-creation-reading-list-cta; fields: none - TK tips at https://w.wiki/FwuD [14:30:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:34:24] 06Data-Engineering, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Task Tries and Logs for Airflow DAGs sometimes unavailable - https://phabricator.wikimedia.org/T419162#12034173 (10Gehel) Yes, this seems to be on our side. >>! In T419162#11698089, @BTullis wrote: > Do we have any way to try to reproduce this... [15:07:05] 06Data-Engineering: event_sanitized.serversideaccountcreation reports users that actually don't exist - https://phabricator.wikimedia.org/T429288#12034366 (10Ottomata) Got it. Welp, I'm not really sure who to report this to. Is there an owner of this legacy ServerSideAccountCreation instrumentation? It is emit... [15:10:45] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12034398 (10JAllemandou) Thank you so much @APizzata-WMF <3 [15:19:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic: Add X-Provenance data to webrequest_sampled_live - https://phabricator.wikimedia.org/T427068#12034446 (10JAllemandou) I have checked the kafka streams, the new field is present in both `webrequest_frontend_text` and `webrequest_sampled`. The bal... [15:23:38] 06Data-Engineering, 06Discovery-Search (2026.06.01 - 2026.07.03), 10Event-Platform, 13Patch-For-Review: Extract eventutilities-flink outside of wikimedia-eventutilities - https://phabricator.wikimedia.org/T429566#12034498 (10dcausse) moved to https://gitlab.wikimedia.org/repos/data-engineering/eventutiliti... [15:29:45] 06Data-Engineering, 06Discovery-Search (2026.06.01 - 2026.07.03), 10Event-Platform, 13Patch-For-Review: Upgrade flink-utilities to flink 2.0.2 - https://phabricator.wikimedia.org/T429565#12034521 (10dcausse) [15:30:07] 06Data-Engineering, 06Discovery-Search (2026.06.01 - 2026.07.03), 10Event-Platform, 13Patch-For-Review: Upgrade flink-utilities to flink 2.0.2 - https://phabricator.wikimedia.org/T429565#12034522 (10dcausse) a:03dcausse [16:00:43] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product-Analytics: dbt-jobs backfill: PP3 API hourly and known clients aggregate jobs - https://phabricator.wikimedia.org/T429341#12034706 (10amastilovic) I've started backfilling `wmf_traffic.mrt_api_requests_hourly` and one day takes about half an hour... [16:02:33] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12034724 (10AKhatun_WMF) Will the `stateless` restart cause any issue like creating or over-writing... [16:08:31] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12034744 (10Ottomata) Hm. The easier thing to do might be to just bump the name of the checkpoint p... [16:37:52] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12034812 (10JMonton-WMF) I have doubts about it, because we checked https://nightlies.apache.org/fl... [16:43:31] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12034815 (10Ottomata) > OffsetsInitializer.earliest() is the default, not OffsetsInitializer.commit... [16:45:24] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12034816 (10JMonton-WMF) That sounds right! after 7 days probably Kafka will remove the `.rc0` offs... [17:08:01] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12034874 (10AKhatun_WMF) From what I am understanding, bumping checkpoint path does the same thing... [17:47:52] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12034989 (10Ottomata) > bumping checkpoint path does the same thing as stateless deployment, withou... [19:36:13] !log Test Kitchen experiment (poll 28489) - adds: donor-delight-badge; removes: none; fields: none - TK tips at https://w.wiki/FwuD [19:36:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [23:13:08] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users level 1 for chudson - https://phabricator.wikimedia.org/T429353#12035851 (10Ladsgroup) According to the note on data.yaml > Approval requests for this group can be expedited by tagging Data-Engineering on phabr...