[03:38:16] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [07:38:16] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [08:14:51] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Observability-Tracing, 13Patch-For-Review: service-utils helper for trace header propagation - https://phabricator.wikimedia.org/T371120#10411702 (10tchin) [09:01:32] (03CR) 10Gehel: Extraction of RawRefineDataReader (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [09:33:46] (03PS20) 10Gehel: Extraction of RawRefineDataReader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 [09:33:46] (03PS3) 10Gehel: refactoring(RefineHelper): extract SparkSchemaLoader from RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1102386 [10:35:27] 06Data-Engineering, 10Structured-Data-Backlog (Current Work): [L] Track commons deletion requests - https://phabricator.wikimedia.org/T370898#10411980 (10Cparle) As of 2024-18-12 --- [[ https://gitlab.wikimedia.org/repos/structured-data/upload-tracking | Code for this ]] is done and in review. To really finis... [11:38:16] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [12:20:44] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Research, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707#10412230 (10BTullis) [13:43:51] 06Data-Engineering, 07Spike: [SPIKE] Evaluate and document solutions for table-management tooling - https://phabricator.wikimedia.org/T360969#10412414 (10Ahoelzl) [13:45:08] 06Data-Engineering, 06Movement-Insights, 07Epic: [Data Quality] Implement basic data quality metrics for Unique Devices datasets - https://phabricator.wikimedia.org/T357833#10412417 (10Ahoelzl) [13:48:47] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 03Discovery-Search (Current work): Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and valida... - https://phabricator.wikimedia.org/T367405#10412418 [13:48:56] 06Data-Engineering: airflow-dags: Mutualization of _IMPORTED flag sensors creations - https://phabricator.wikimedia.org/T371373#10412419 (10Ahoelzl) [13:50:57] (03PS9) 10Peter Fischer: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) [13:52:35] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform, 10Dumps-Generation: 20241201 wikidatawiki xml dump not progressing - https://phabricator.wikimedia.org/T382084#10412438 (10Ahoelzl) [13:53:22] 06Data-Engineering, 06Research: Incremental HTML dataset to support "Who are moderators" SDS 1.2.3 - https://phabricator.wikimedia.org/T380874#10412441 (10Ahoelzl) [13:54:42] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Figure why XML Dump code generates 1000's of files for simplewiki - https://phabricator.wikimedia.org/T381016#10412443 (10pfischer) First iteration successfully dumped `simplewiki` but failed to do so on... [13:56:25] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10CheckUser, 06Data Products, 06DBA, and 2 others: Remove cuc_actiontext, cuc_only_for_read_old, and cuc_private from cu_changes on WMF wikis - https://phabricator.wikimedia.org/T370903#10412449 (10Ahoelzl) [13:57:34] 10Data-Engineering (Q2 2024 October 1st - December 31th): [SPIKE] Define process to build out lineage in DataHub - https://phabricator.wikimedia.org/T369758#10412461 (10Ahoelzl) 05Open→03Resolved a:03Ahoelzl [14:21:23] (03CR) 10Jforrester: "recheck" [analytics/mediawiki-storage] - 10https://gerrit.wikimedia.org/r/681176 (owner: 10Milimetric) [14:25:44] heh, go figure, I didn't think that would rebuild :) [15:38:16] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [16:15:37] (03PS5) 10Mforns: Edit Geoeditors Daily Monthly to support Temp Account Changes [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1098083 (https://phabricator.wikimedia.org/T379728) (owner: 10Jennifer Ebe) [16:33:22] (03PS10) 10Peter Fischer: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) [16:53:17] (03CR) 10CI reject: [V:04-1] Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) (owner: 10Peter Fischer) [17:30:19] 10Data-Engineering (Q2 2024 October 1st - December 31th): Gitlab CI/CD Component for Blunderbuss - https://phabricator.wikimedia.org/T382348#10413220 (10amastilovic) @Ottomata I've addressed your concerns re: number of repositories in this ticket: https://phabricator.wikimedia.org/T382430 [17:55:39] 06Data-Engineering: Create a GitLab CI/CD Component project for WMF CI/CD templates and components - https://phabricator.wikimedia.org/T382430#10413377 (10Aklapper) [19:15:40] 06Data-Engineering, 10BDC-Implementation, 06Data-Platform, 06Data-Platform-SRE, 07Epic: EPIC: Trino/minIO/Hive/Hadoop Implementation - https://phabricator.wikimedia.org/T377362#10413790 (10Irisbdc) [19:24:34] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 13Patch-For-Review: Deploy the HDFS synchronizer (blunderbuss) service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10413814 (10amastilovic) Helm charts: https://gerrit.wikimedia... [19:24:47] 10Data-Engineering (Q2 2024 October 1st - December 31th): Implement automated deployment of refinery HQL files to HDFS (via blunderbuss) - https://phabricator.wikimedia.org/T365659#10413815 (10amastilovic) [19:38:16] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [20:29:06] 06Data-Engineering, 10LPL Analytics, 06Product-Analytics, 10Event-Platform: Invalid EventGate errors with content_translation_event 1.7.0 - https://phabricator.wikimedia.org/T382113#10414027 (10Ottomata) [20:30:32] 06Data-Engineering, 10LPL Analytics, 06Product-Analytics, 10Event-Platform: Invalid EventGate errors with content_translation_event 1.7.0 - https://phabricator.wikimedia.org/T382113#10414038 (10Ottomata) Done. @KCVelaga_WMF how's it look now? [20:32:53] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: EventBus PageChangeHooks uses unconventional log channel name - https://phabricator.wikimedia.org/T382288#10414044 (10Ottomata) a:03Ottomata [20:32:56] 06Data-Engineering, 10LPL Analytics, 06Product-Analytics, 10Event-Platform: Invalid EventGate errors with content_translation_event 1.7.0 - https://phabricator.wikimedia.org/T382113#10414045 (10Ottomata) a:03Ottomata [20:33:24] 06Data-Engineering, 10LPL Analytics, 06Product-Analytics, 10Event-Platform: Invalid EventGate errors with content_translation_event 1.7.0 - https://phabricator.wikimedia.org/T382113#10414051 (10Ottomata) [20:34:06] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: EventBus PageChangeHooks uses unconventional log channel name - https://phabricator.wikimedia.org/T382288#10414064 (10Ottomata) [20:34:40] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: EventBus PageChangeHooks uses unconventional log channel name - https://phabricator.wikimedia.org/T382288#10414077 (10Ottomata) [20:49:41] 10Data-Engineering (Q2 2024 October 1st - December 31th): Gitlab CI/CD Component for Blunderbuss - https://phabricator.wikimedia.org/T382348#10414108 (10Ottomata) Thank you! [20:51:08] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 13Patch-For-Review: Deploy the HDFS synchronizer (blunderbuss) service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10414109 (10Ottomata) Thank you for the rename! [21:11:14] 06Data-Engineering, 10Structured-Data-Backlog (Current Work): [L] Track commons deletion requests - https://phabricator.wikimedia.org/T370898#10414173 (10amastilovic) Hey @Cparle , @Ottomata tells me you might need HDFS deployment through CI/CD for this project? I just set up Blunderbuss which does exactly tha... [21:22:24] 06Data-Engineering, 06Data Products: [refine] Add support for custom Hive partitioning - https://phabricator.wikimedia.org/T377600#10414196 (10Ottomata) [22:02:28] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data Products, 06Data-Platform, 06Movement-Insights, 13Patch-For-Review: Backfill and recalculate unique devices data from July 2024 to present - https://phabricator.wikimedia.org/T378852#10414303 (10Hghani) Thank you all for the quick turn arou... [23:38:16] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent