[00:29:06] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Revert webrequest retention time back to 90 days - https://phabricator.wikimedia.org/T412321 (10Ahoelzl) 03NEW [00:32:26] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Investigate and ensure Druid snapshot retention is limited to 3 - https://phabricator.wikimedia.org/T412322 (10Ahoelzl) 03NEW [11:02:26] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06MW-Interfaces-Team, 10Event-Platform: mediawiki.page_change.v1 event stream - Investigate mistmatched meta.dt and dt (and rev_dt) fields - https://phabricator.wikimedia.org/T409105#11450882 (10JMonton-WMF) I think this might be a bit more complex... [14:57:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform, 07Spike: Explore how to migrate PyFlink to Java/Scala - https://phabricator.wikimedia.org/T410266#11451778 (10JMonton-WMF) After some conversations, we think there are different paths related to this: - How we support other teams... [15:19:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:19:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:07:32] 06Data-Engineering, 06Data-Engineering-Radar, 10GrowthExperiments, 06MediaWiki-Engineering, and 7 others: mw.track: support for histogram metrics - https://phabricator.wikimedia.org/T383563#11452154 (10Michael) @Krinkle @colewhite I've added you both as reviewers for the `performance/statsv` change needed... [17:40:43] 06Data-Engineering, 10Dumps-Generation, 13Patch-For-Review: Update dump mirror rsync allowlist to reflect new IP address for Scatter - https://phabricator.wikimedia.org/T409006#11452343 (10xcollazo) Confirmed via contact email that the request is legit. [17:52:50] 06Data-Engineering, 10Dumps-Generation, 10Wikidata data dumps, 07Wikimedia-production-error: Wikidata full .json.gz dumps not published since 20250625 - https://phabricator.wikimedia.org/T412428#11452363 (10Pppery) a:05Andrew→03None [17:58:47] 06Data-Engineering, 10Dumps-Generation, 10Wikidata data dumps: Wikidata full .json.gz dumps not published since 20250625 - https://phabricator.wikimedia.org/T412428#11452397 (10taavi) [19:01:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Infrastructure-Foundations, 10netops: Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443 (10JAllemandou) 03NEW [19:19:51] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:19:51] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:22:38] !log Test Kitchen edge-unique experiments (poll 8600) - adds: none; removes: sticky-headers; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [19:22:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:36:19] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Infrastructure-Foundations, 10netops: Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11452701 (10ayounsi) Thanks, basically there was a firewall rule that prevented the ingestion of the SFLOW traffic from... [19:39:18] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Infrastructure-Foundations, 10netops: Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11452708 (10cmooney) @JAllemandou thanks for the task. And apologies this has just hit you out of the blue - we should... [20:03:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Infrastructure-Foundations, 10netops: Handle `network_flows_internal` data growth - https://phabricator.wikimedia.org/T412443#11452763 (10xcollazo) ( For now, we have paused the pipeline, as it is continually failing: https://airflow.wikimedia.or... [20:15:14] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Wikidata data dumps: Wikidata full .json.gz dumps not published since 20250625 - https://phabricator.wikimedia.org/T412428#11452808 (10xcollazo) `20251210` is available at https://dumps.wikimedia.org/other/wikibase/wikidatawiki/20251210/. Perhaps an N... [21:21:04] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Increase partitions of mediawiki.content_history_reconcile.v1 - https://phabricator.wikimedia.org/T411598#11453042 (10xcollazo) (Side note: I just bumped TaskManagers to 20 temporarily due to T411803). [21:24:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:24:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:25:48] 06Data-Engineering, 10Dumps-Generation, 13Patch-For-Review: Update dump mirror rsync allowlist to reflect new IP address for Scatter - https://phabricator.wikimedia.org/T409006#11453054 (10xcollazo) @Harej please confirm whether `rsync` works on your end. [21:33:27] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Wikidata data dumps: Wikidata full .json.gz dumps not published since 20250625 - https://phabricator.wikimedia.org/T412428#11453088 (10Andrew) This is likely related to the refactoring done for T352650. WMCS staff is traveling this week so before I dig... [21:42:06] 06Data-Engineering, 10DPE-Mediawiki-Content: On reconcile, consider what happens when a restore and a delete happen on the same revision - https://phabricator.wikimedia.org/T412461 (10xcollazo) 03NEW [21:42:22] 06Data-Engineering, 10DPE-Mediawiki-Content: On reconcile, consider what happens when a restore and a delete happen on the same revision - https://phabricator.wikimedia.org/T412461#11453122 (10xcollazo) [21:42:22] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps (SDS 1.2) - https://phabricator.wikimedia.org/T366752#11453123 (10xcollazo) [22:02:22] !log Test Kitchen edge-unique experiments (poll 9076) - adds: sticky-headers; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:02:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:58:54] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Fix reconcile bug where user_id is not being populated correctly. - https://phabricator.wikimedia.org/T411803#11453273 (10xcollazo) There are a lot of reconcile events to be ingested (See [[ https://grafana.wikimedia.org/d/K9x0c4a...