[00:42:06] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Jupyter, 06Product-Analytics: Functionality to share & view notebooks - https://phabricator.wikimedia.org/T156934#10746042 (10nshahquinn-wmf) 05Open→03Declined I'm boldly declining this. We have a reasonable set of tools for sharing... [01:15:35] 06Data-Engineering, 10Data-Engineering-Jupyter: Conda-Analytics has package conflict when trying to install R with key packages (R-Arrow and R-Stringi) - https://phabricator.wikimedia.org/T391911#10746074 (10nshahquinn-wmf) I've actually found a better workaround (the Nanoparquet package—details in the descrip... [01:26:00] 06Data-Engineering, 10Data-Engineering-Jupyter: Conda-Analytics has package conflict when trying to install R with key packages (R-Arrow and R-Stringi) - https://phabricator.wikimedia.org/T391911#10746104 (10nshahquinn-wmf) I've documented this issue and the workaround on https://wikitech.wikimedia.org/wiki/Da... [06:20:46] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for Lena Meintrup - https://phabricator.wikimedia.org/T391820#10746393 (10Lena_WMDE) @MatthewVernon works as expected, thank you! :) [06:36:21] 06Data-Engineering, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10746399 (10Volans) As reported on the parent task we will create a new host with bookworm and keep... [07:26:27] 06Data-Engineering, 06Traffic, 10DPE HAProxy Migration, 13Patch-For-Review: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10746435 (10Fabfur) @JAllemandou I've prepared [[ https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/82 | this patch ]] for h... [07:45:10] 10Data-Engineering-Jupyter, 06Data-Platform-SRE: Upgrade to JupyterLab ≥ 4.2 - https://phabricator.wikimedia.org/T391905#10746496 (10Gehel) p:05Triage→03Medium [07:48:25] 06Data-Engineering, 10Data-Platform-SRE (2025-04-12 - 2025-05-02), 07Documentation: https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log should be on Wikitech - https://phabricator.wikimedia.org/T387878#10746508 (10Gehel) p:05Triage→03Medium [10:07:15] (03PS2) 10KCVelaga: Add MinT for Readers stream to sanitization allow list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1133484 (https://phabricator.wikimedia.org/T372724) [10:09:34] (03CR) 10Lucas Werkmeister (WMDE): Add Prometheus stats push (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [10:09:37] (03CR) 10KCVelaga: "@mforns@wikimedia.org Please add your vote again (removed the extra space), and please merge as well if it looks good you. Thank you." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1133484 (https://phabricator.wikimedia.org/T372724) (owner: 10KCVelaga) [10:16:04] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop afl_patrolled_by from abuse_filter_log in production - https://phabricator.wikimedia.org/T391056#10746770 (10FCeratto-WMF) [10:22:37] (03CR) 10Lucas Werkmeister (WMDE): Add Prometheus stats push (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [10:41:59] (03CR) 10Filippo Giunchedi: "I left some comments inline" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [10:52:31] (03CR) 10Lucas Werkmeister (WMDE): Add Prometheus stats push (033 comments) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [11:02:05] (03CR) 10Lucas Werkmeister (WMDE): "> The naming convention for Prometheus labels and metric names is with underscores not camelCase" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [11:02:05] (03PS9) 10Lucas Werkmeister (WMDE): Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [12:29:19] (03PS10) 10Lucas Werkmeister (WMDE): Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [12:29:57] (03CR) 10Lucas Werkmeister (WMDE): "Made some more changes that hopefully make sense, esp. renaming a few metrics away from `_count_total` to just `_count`. But I also have t" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [13:45:16] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: Epic: MinIO implementation - https://phabricator.wikimedia.org/T392090 (10Jgreen) 03NEW [13:45:41] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: Epic: MinIO implementation - https://phabricator.wikimedia.org/T392090#10747662 (10Jgreen) [13:47:04] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: Epic: MinIO implementation - https://phabricator.wikimedia.org/T392090#10747669 (10Jgreen) [13:47:35] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation - https://phabricator.wikimedia.org/T377362#10747674 (10Jgreen) [13:48:08] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: MinIO implementation - https://phabricator.wikimedia.org/T392090#10747677 (10Jgreen) a:05greg→03None [13:49:35] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Canary failure on airflow platform_eng intsance after migrating to Kubernetes - https://phabricator.wikimedia.org/T390727#10747685 (10brouberol) 05In progress→03Resolved [13:49:47] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: MinIO implementation - https://phabricator.wikimedia.org/T392090#10747688 (10Jgreen) [13:49:50] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 10fundraising-tech-ops, 07Epic: TLS connection for hive-standalone-metaserver with minio - https://phabricator.wikimedia.org/T385031#10747689 (10Jgreen) [13:51:40] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: MinIO implementation - https://phabricator.wikimedia.org/T392090#10747710 (10Jgreen) [13:53:24] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino implementation - https://phabricator.wikimedia.org/T392093 (10Jgreen) 03NEW [13:54:15] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino implementation - https://phabricator.wikimedia.org/T392093#10747753 (10Jgreen) a:05greg→03None [13:58:07] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino implementation - https://phabricator.wikimedia.org/T392093#10747767 (10Jgreen) [13:58:10] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, and 2 others: [Trino] Develop procedure and scripting for Trino cluster maintenance. - https://phabricator.wikimedia.org/T386391#10747769 (10Jgreen) [13:58:59] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation - https://phabricator.wikimedia.org/T377362#10747783 (10Jgreen) [13:59:05] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 10fundraising-tech-ops, 07Epic: TLS connection for hive-standalone-metaserver with minio - https://phabricator.wikimedia.org/T385031#10747782 (10Jgreen) [13:59:19] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation - https://phabricator.wikimedia.org/T377362#10747787 (10Jgreen) [13:59:37] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation - https://phabricator.wikimedia.org/T377362#10747789 (10Jgreen) [14:00:00] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation - https://phabricator.wikimedia.org/T377362#10747793 (10Jgreen) [14:06:47] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation - https://phabricator.wikimedia.org/T377362#10747848 (10Jgreen) [14:09:23] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino implementation - https://phabricator.wikimedia.org/T392093#10747872 (10Jgreen) [14:13:45] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, and 2 others: [Trino] Develop procedure and scripting for Trino cluster maintenance. - https://phabricator.wikimedia.org/T386391#10747901 (10Jgreen) [14:13:46] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation - https://phabricator.wikimedia.org/T377362#10747902 (10Jgreen) [14:22:02] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: MinIO implementation - https://phabricator.wikimedia.org/T392090#10747986 (10Jgreen) [14:22:03] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino implementation - https://phabricator.wikimedia.org/T392093#10747985 (10Jgreen) [14:22:05] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation - https://phabricator.wikimedia.org/T377362#10747987 (10Jgreen) [14:25:53] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino implementation - https://phabricator.wikimedia.org/T392093#10748031 (10Jgreen) [14:33:48] (03CR) 10Filippo Giunchedi: "Ah yes I think you are right, I forgot these are technically mw metrics. IIRC for metric names in MW components we're doing camelCase thou" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [14:54:11] 10Quarry: Quarry down? - https://phabricator.wikimedia.org/T392107 (10Alien333) 03NEW [14:56:40] 10Quarry: Quarry down? - https://phabricator.wikimedia.org/T392107#10748247 (10Alien333) [15:06:36] 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10748309 (10Aklapper) [15:08:08] 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10748324 (10Alien333) [15:19:42] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: [MinIO] Improve cluster to minimum system configuration for production - https://phabricator.wikimedia.org/T392112 (10Jgreen) 03NEW [15:20:39] 10Data-Engineering (Q4 2025 April 1st - June 30th), 07Essential-Work: Support for 4.3.11 - webrequest based scraping detection - https://phabricator.wikimedia.org/T388721#10748380 (10Ahoelzl) On hold until approach is reviewed. [15:28:39] (03PS11) 10Lucas Werkmeister (WMDE): Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [15:29:14] (03PS12) 10Lucas Werkmeister (WMDE): Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [15:29:41] (03CR) 10Lucas Werkmeister (WMDE): "For now I just went for throwing an error anyway, because we probably don’t need `"` or `\` characters (nor `'` in any part of the labels," [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [15:32:52] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: [MinIO] Improve cluster to minimum system configuration for production - https://phabricator.wikimedia.org/T392112#10748435 (10Jgreen) [15:56:43] 10Data-Engineering (Q4 2025 April 1st - June 30th): NEW BUG REPORT significantly increased edit revert rate for 2025-03 edits; Android, iOS, Mobile Web, Other - https://phabricator.wikimedia.org/T391708#10748576 (10Ahoelzl) a:03mforns [17:21:19] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: [Trino] additional worker nodes for eqiad - https://phabricator.wikimedia.org/T392131 (10Jgreen) 03NEW [17:53:12] 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10749174 (10SD0001) From the logs: ` redis.exceptions.ResponseError: MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk. Commands that may modify the data s... [17:55:08] 10Quarry, 06cloud-services-team: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10749199 (10bd808) p:05Triage→03High [18:06:52] 10Quarry, 06cloud-services-team: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10749254 (10SD0001) Redis RDB persistence is failing as the pod is out of disk space. ` 4134:C 16 Apr 2025 17:53:32.082 # Failed opening the temp RDB file temp-4134.rdb (in ser... [18:17:08] 10Quarry, 06cloud-services-team: No alerting for quarry - https://phabricator.wikimedia.org/T392138 (10Andrew) 03NEW [18:50:07] 10Quarry, 06cloud-services-team: Update quarry redis deployment - https://phabricator.wikimedia.org/T392141 (10Andrew) 03NEW [18:51:12] 10Quarry, 06cloud-services-team: Quarry: Why so many web pods? - https://phabricator.wikimedia.org/T392143 (10Andrew) 03NEW [18:57:14] 10Quarry, 06cloud-services-team: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10749506 (10Andrew) 05Open→03Resolved a:03Andrew This seems to have been a disk space issue on one of the worker nodes. I rebooted both nodes, and then taavi killed ex... [23:58:42] 10Quarry: [bug] Quarry queries don't run - https://phabricator.wikimedia.org/T392169 (10Liz) 03NEW