[09:10:07] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10CirrusSearch, 03Discovery-Search (Current work): [Search Update Pipeline] Source streams for private wikis - https://phabricator.wikimedia.org/T346046#10068697 (10Gehel) 05Open→03Resolved [09:15:13] 06Data-Engineering, 03Discovery-Search (Current work), 10MW-1.43-notes (1.43.0-wmf.17; 2024-08-06), 07Wikimedia-production-error: '.event.pageViewId' should be string, '.event.subTest' should be string, '.event.searchSessionId' should be string - https://phabricator.wikimedia.org/T286814#10068708 (10Geh... [09:17:37] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Rollback haproxy feed automated ingestion - https://phabricator.wikimedia.org/T372456#10068725 (10gmodena) [09:23:44] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): request for new matomo site: trace.wikimedia.org/ - https://phabricator.wikimedia.org/T371124#10068728 (10Gehel) 05Open→03Resolved @CDanis : I'm closing this as it seems completed from our side. Please re-open and ping me on Slack if yo... [09:43:17] 06Data-Engineering, 10Data-Platform-SRE (2024.08.17 - 2024.09.06): Reset kerberos password for WMDE-leszek - https://phabricator.wikimedia.org/T365137#10068839 (10Gehel) [09:44:21] 06Data-Engineering, 10Data-Platform-SRE (2024.08.17 - 2024.09.06): Requesting Kerberos access for ifrahkhanyaree - https://phabricator.wikimedia.org/T371894#10068862 (10Gehel) [09:45:39] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 10Data-Platform-SRE (2024.08.17 - 2024.09.06): Create puppet defined type for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#10068874 (10Gehel) [09:47:08] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.08.17 - 2024.09.06): Update parent pom to disable fetching dependencies from Archiva and use Gitlab instead - https://phabricator.wikimedia.org/T367404#10068886 (10Gehel) [09:50:21] 06Data-Engineering, 10Data-Platform-SRE (2024.08.17 - 2024.09.06): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10068928 (10Gehel) [09:50:35] 06Data-Engineering, 10Data-Platform-SRE (2024.08.17 - 2024.09.06): Some wikibase tables not available in commonswiki_p - https://phabricator.wikimedia.org/T298452#10068932 (10Gehel) [09:51:20] 06Data-Engineering, 10Data-Platform-SRE (2024.08.17 - 2024.09.06): an-launcher1002 /srv filling up mostly because of logs from dynamic mapped Airflow tasks - https://phabricator.wikimedia.org/T370437#10068952 (10Gehel) [09:51:48] 06Data-Engineering, 06Discovery-Search, 10Data-Platform-SRE (2024.08.17 - 2024.09.06), 07IPv6: Some Search clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts - https://phabricator.wikimedia.org/T312555#10068957 (10Gehel) [14:07:17] 06Data-Engineering: Handle Late-Arrived Events from Gobblin into Airflow triggered Refine - https://phabricator.wikimedia.org/T370665#10069544 (10Ottomata) > The first one is a custom sensor that succeeds when the source partition has seen an update (check file mtime?) > The second task is a custom operator that... [14:11:28] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Airflow RestExternalTaskSensor should be able to sense named dynamic mapped tasks - https://phabricator.wikimedia.org/T372644 (10Ottomata) 03NEW [14:28:11] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647 (10Ottomata) 03NEW [14:29:10] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10069622 (10Ottomata) [14:29:59] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10069623 (10Ottomata) [14:30:56] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10069626 (10Ottomata) [14:51:22] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.19; 2024-08-20), 13Patch-For-Review: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#10069685 (10Ottomata) Just te... [14:57:10] 10Data-Engineering (Q1 2024 July 1st - September 30th): Airflow RestExternalTaskSensor should be able to sense named dynamic mapped tasks - https://phabricator.wikimedia.org/T372644#10069716 (10Ottomata) [14:58:05] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10069718 (10Ottomata) [15:05:04] 06Data-Engineering, 10Data-Platform-SRE (2024.08.17 - 2024.09.06): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10069740 (10Gehel) [15:19:41] FIRING: MediawikiPageContentChangeEnrichAvailability: ... [15:19:41] Low percentage of enriched events produced by mw_page_content_change_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability [15:42:20] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10069856 (10mforns) Maybe just change the name of the file to `data_dependencies.yaml` and the module to `DataDependency`? A... [16:03:41] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10069897 (10Ottomata... [16:11:43] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10069934 (10Ottomata) Ya probably some renaming like you suggest would remove the awkwardness. Looking forward to the namin... [16:41:27] 06Data-Engineering, 10Data-Platform-SRE (2024.08.17 - 2024.09.06): Requesting Kerberos access for ifrahkhanyaree - https://phabricator.wikimedia.org/T371894#10070091 (10BTullis) This does seem odd. The next thing to do is to verify that the key you have selected for use with this SSH connection is the same one... [18:22:02] 06Data-Engineering, 10Dumps 2.0 (Kanban Board): [Iceberg Migration] Implement mechanism for automatic Iceberg data deletion and optimization - https://phabricator.wikimedia.org/T338065#10070292 (10xcollazo) 05Open→03In progress p:05Triage→03Medium a:03xcollazo [18:24:16] 06Data-Engineering, 10Dumps 2.0 (Kanban Board): [Iceberg Migration] Implement mechanism for automatic Iceberg data deletion and optimization - https://phabricator.wikimedia.org/T338065#10070298 (10xcollazo) Started working on {T358365}, and figured it would be silly to implement this just for #dumps_2.0 . So... [18:25:41] 06Data-Engineering, 10Dumps 2.0 (Kanban Board): [Iceberg Migration] Implement mechanism for automatic Iceberg data deletion and optimization - https://phabricator.wikimedia.org/T338065#10070307 (10xcollazo) [19:19:56] FIRING: MediawikiPageContentChangeEnrichAvailability: ... [19:19:56] Low percentage of enriched events produced by mw_page_content_change_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability [19:59:20] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Dumps 2] Spike: Figure root causes of missing rows when doing reconciliation - https://phabricator.wikimedia.org/T368176#10070512 (10xcollazo) 05Open→03Resolved [22:06:38] 10Data-Engineering (Q1 2024 July 1st - September 30th): [Data Quality] Improve Superset visualizations - https://phabricator.wikimedia.org/T372678 (10Ahoelzl) 03NEW [23:19:56] FIRING: MediawikiPageContentChangeEnrichAvailability: ... [23:19:56] Low percentage of enriched events produced by mw_page_content_change_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability