[01:45:42] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review, 07Verified, 10Web Team Essential Work 2024 (Migrate to new Event Platform): Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065#10302617 (10Edtadros) [02:50:17] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: Suppress tracebacks for Kerberos errors - https://phabricator.wikimedia.org/T345219#10302648 (10nshahquinn-wmf) p:05Triage→03Low [02:55:26] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: Add utility functions for converting between various time formats - https://phabricator.wikimedia.org/T273209#10302651 (10nshahquinn-wmf) [03:00:27] 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python, 10GitLab (Project Migration), 13Patch-For-Review: Move Wmfdata-Python from Github to Gitlab - https://phabricator.wikimedia.org/T304544#10302652 (10nshahquinn-wmf) >>! In T304544#10291320, @CodeReviewBot wrote: > nshahquinn-wmf opened https://git... [03:07:07] 06Data-Engineering, 10Wmfdata-Python: Some Wmfdata docstrings are not recognized by Python - https://phabricator.wikimedia.org/T379339 (10nshahquinn-wmf) 03NEW p:05Triage→03Medium [03:07:54] 06Data-Engineering, 10Wmfdata-Python: Some Wmfdata docstrings are not recognized by Python - https://phabricator.wikimedia.org/T379339#10302666 (10nshahquinn-wmf) First step is for someone else to verify that they can reproduce this with their installation. [03:24:30] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: Remove "master" terminology from wmfdata-python - https://phabricator.wikimedia.org/T272220#10302673 (10nshahquinn-wmf) 05Stalled→03Resolved a:03nshahquinn-wmf At this point, it doesn't seem like Spark is going to change their use of the te... [03:30:51] 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python, 07Documentation: Public HTML docs for Wmfdata-Python on doc.wikimedia.org - https://phabricator.wikimedia.org/T298178#10302678 (10nshahquinn-wmf) [04:06:58] 06Data-Engineering, 10Wmfdata-Python: Wmfdata: Spark session creation fails when there is a zombie session - https://phabricator.wikimedia.org/T367998#10302699 (10nshahquinn-wmf) p:05Triage→03Low [09:55:26] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Deploy the HDFS synchronizer (blunderbuss) service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10303051 (10Gehel) [09:56:58] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 10Data-Platform-SRE (2024.11.09 - 2024.11.29), 13Patch-For-Review: Create puppet resource for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#10303049 (10Gehel) [09:57:21] 06Data-Engineering, 10Prod-Kubernetes, 06serviceops, 10Data-Platform-SRE (2024.11.09 - 2024.11.29), and 3 others: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10303057 (10Gehel) [09:59:48] 06Data-Engineering, 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Some wikibase tables not available in commonswiki_p - https://phabricator.wikimedia.org/T298452#10303083 (10Gehel) [10:00:31] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29), 13Patch-For-Review: Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10303093 (10Gehel) [10:00:33] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Discovery-Search, 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Upload an image with flink-k8s-operator version that supports flink 1.20 - https://phabricator.wikimedia.org/T377137#10303097 (10Gehel) [10:01:24] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Discovery-Search, 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Create and distribute a flink base image with flink 1.20.0 - https://phabricator.wikimedia.org/T377134#10303095 (10Gehel) [10:01:36] 06Data-Engineering, 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10303115 (10Gehel) [10:03:07] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Refine jobs fail to send alert emails due to a decommissioned MX - https://phabricator.wikimedia.org/T377698#10303150 (10Gehel) [10:03:17] 06Data-Engineering, 06Data-Platform-SRE: Implement periodical cleaning of Airflow databases - https://phabricator.wikimedia.org/T322036#10303152 (10brouberol) 05Open→03Resolved a:03brouberol Resolved by https://phabricator.wikimedia.org/T375878 [10:03:27] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users group, sql_lab role, Kerberos Principal for Khantstop - https://phabricator.wikimedia.org/T379303#10303169 (10MatthewVernon) [10:04:59] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users group, sql_lab role, Kerberos Principal for Khantstop - https://phabricator.wikimedia.org/T379303#10303183 (10MatthewVernon) [10:10:41] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users group, sql_lab role, Kerberos Principal for Khantstop - https://phabricator.wikimedia.org/T379303#10303187 (10MatthewVernon) Hi @Khantstop, I think you're a contractor - can you or @OSefu-WMF confirm the contra... [10:17:21] 06Data-Engineering, 06Structured-Data-Backlog: Remove event.suggestedtagsaction from hive - https://phabricator.wikimedia.org/T360994#10303233 (10BTullis) [10:21:58] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10303242 (10BTullis) [10:41:15] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Refine jobs fail to send alert emails due to a decommissioned MX - https://phabricator.wikimedia.org/T377698#10303309 (10BTullis) @gmodena Are we able to resolve this now, or is there anything outstanding t... [10:45:48] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Refine jobs fail to send alert emails due to a decommissioned MX - https://phabricator.wikimedia.org/T377698#10303332 (10gmodena) @BTullis this was closed on the DE board, and IMHO we can resolve it. From w... [10:46:36] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Refine jobs fail to send alert emails due to a decommissioned MX - https://phabricator.wikimedia.org/T377698#10303335 (10BTullis) 05Open→03Resolved Great! Thanks. [11:00:06] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10303370 (10BTullis) p:05Medium→03High [14:01:07] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29), 13Patch-For-Review: Update druid config to automatically drop unused segments - https://phabricator.wikimedia.org/T376118#10303786 (10BTullis) a:03BTullis >>! In T376118#10216466, @JAllemandou wrote: >>... [14:43:19] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users group, sql_lab role, Kerberos Principal for Khantstop - https://phabricator.wikimedia.org/T379303#10304054 (10MatthewVernon) 05Open→03Resolved a:03MatthewVernon Done, and Kerbero... [15:18:28] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform, 06Traffic: NEW BUG REPORT - Issues in calculation logic for unique devices tables - https://phabricator.wikimedia.org/T375527#10304221 (10OSefu-WMF) @VirginiaPoundstone I see that this task was 'resolved' but as a reminder, there... [15:52:44] 06Data-Engineering, 06Data-Platform-SRE, 07Epic: Plan for a Hadoop and Hive upgrade for the Data Platform - https://phabricator.wikimedia.org/T379385 (10BTullis) 03NEW [16:04:43] 10Data-Engineering (Q2 2024 October 1st - December 31th), 03Discovery-Search (Current work), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Add support for Spark producers in Event Platform - https://phabricator.wikimedia.org/T374341#10304460 (10xcollazo) 05Open→03In progress [16:41:06] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Deploy the HDFS synchronizer (blunderbuss) service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10304616 (10bking) [16:41:27] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Deploy the HDFS synchronizer (blunderbuss) service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10304619 (10bking) [16:59:43] 10Data-Engineering (Q2 2024 October 1st - December 31th), 13Patch-For-Review: [HAProxy transition] Deploy a staging airflow dag for webrequest refinement - https://phabricator.wikimedia.org/T378342#10304716 (10Ahoelzl) [17:12:22] 06Data-Engineering, 10Prod-Kubernetes, 06serviceops, 10Data-Platform-SRE (2024.11.09 - 2024.11.29), and 3 others: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10304743 (10bking) [17:14:01] 06Data-Engineering, 10Prod-Kubernetes, 06serviceops, 10Data-Platform-SRE (2024.11.09 - 2024.11.29), and 3 others: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10304746 (10bking) 05Open→03Resolved I just crossed out the `flink-opera... [17:24:51] 06Data-Engineering, 06Data-Platform-SRE: SPIKE - Will Hadoop 3 container support help us for Airflow deployment pipelines? - https://phabricator.wikimedia.org/T288247#10304776 (10BTullis) [17:24:52] 06Data-Engineering, 06Data-Platform-SRE, 07Epic: Plan for a Hadoop and Hive upgrade for the Data Platform - https://phabricator.wikimedia.org/T379385#10304777 (10BTullis) [17:26:07] 06Data-Engineering, 06Data-Platform-SRE: SPIKE - Will Hadoop 3 container support help us for Airflow deployment pipelines? - https://phabricator.wikimedia.org/T288247#10304773 (10BTullis) a:03BTullis I'm going to claim this task because I'm investigating container support as part of a Hadoop 3 upgrade plan. [17:40:52] 06Data-Engineering, 06Data-Platform-SRE, 07Epic: Plan for a Hadoop and Hive upgrade for the Data Platform - https://phabricator.wikimedia.org/T379385#10304807 (10xcollazo) >Will we continue to use bigtop packages, or should we look into some form of containerised approach? The commercial world of Hadoop dis... [17:54:31] 06Data-Engineering, 06Data-Platform-SRE: SPIKE - Will Hadoop 3 container support help us for Airflow deployment pipelines? - https://phabricator.wikimedia.org/T288247#10304874 (10BTullis) Since this ticket was originally created, the guidance around using the docker daemon in production has been [[https://wiki... [17:57:26] 06Data-Engineering, 10Research-engineering, 06Research-Freezer, 10Event-Platform: [Research Engineering Request] Productionized Edit Types - https://phabricator.wikimedia.org/T351225#10304886 (10diego) [17:58:42] 06Data-Engineering, 10Event-Platform: Implement stream of HTML content on mw.page_change event - https://phabricator.wikimedia.org/T360794#10304889 (10diego) [18:15:02] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Metrics Platform, 10MediaWiki-Platform-Team (Radar): Cannot install wikimedia/timestamp 4.2.0 due to conflicting wikimedia/metrics-platform requiring wikimedia/timestamp 4.1.1 - https://phabricator.wikimedia.org/T378275#10304963 (10Reedy) [18:39:30] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users group, sql_lab role, Kerberos Principal for Khantstop - https://phabricator.wikimedia.org/T379303#10305066 (10Khantstop) @MatthewVernon are you able to add me to sql_lab role as well? I don't think this was... [18:51:02] 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python, 10GitLab (Project Migration), 13Patch-For-Review: Move Wmfdata-Python from Github to Gitlab - https://phabricator.wikimedia.org/T304544#10305108 (10nshahquinn-wmf) >>! In T304544#10304894, @CodeReviewBot wrote: > nshahquinn-wmf opened https://git... [19:28:37] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: `spark.memory.driver` option does not get applied with "client" deployment mode. - https://phabricator.wikimedia.org/T284630#10305245 (10nshahquinn-wmf) 05Open→03Resolved a:03nshahquinn-wmf This seems to be fixed now. I tried the test case... [19:45:12] 06Data-Engineering, 10Wmfdata-Python: Some Wmfdata docstrings are not recognized by Python - https://phabricator.wikimedia.org/T379339#10305322 (10nshahquinn-wmf) 05Open→03Invalid I've tried this in a fresh environment and everything works fine, so I think this was just a weird one-off issue in my envi... [20:01:30] 06Data-Engineering, 10Wmfdata-Python: Separate unit tests and integration tests - https://phabricator.wikimedia.org/T379416 (10nshahquinn-wmf) 03NEW p:05Triage→03Medium [20:16:47] 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python, 07Documentation: Publish HTML docs for Wmfdata-Python on doc.wikimedia.org - https://phabricator.wikimedia.org/T298178#10305417 (10nshahquinn-wmf) [20:27:06] 06Data-Engineering, 06Movement-Insights, 10Wmfdata-Python, 07Documentation: Publish HTML docs for Wmfdata-Python on doc.wikimedia.org - https://phabricator.wikimedia.org/T298178#10305463 (10nshahquinn-wmf) I've just changed the plan: previously, it was to use [Pdoc](https://pdoc.dev/), but now it's to use... [20:41:53] 06Data-Engineering, 06Research, 10Research-engineering, 10Event-Platform: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#10305494 (10leila) p:05Low→03Triage [20:44:02] 06Data-Engineering, 10Research-engineering, 10Event-Platform, 10Research (FY2024-25-Research-January-March): Productionized Edit Types - https://phabricator.wikimedia.org/T351225#10305514 (10leila) This task (or a version of it to be updated later in Q2) is now a blocker for T371865 (work on "who are moder... [20:46:20] 06Data-Engineering, 10Research-engineering, 10Event-Platform, 10Research (FY2024-25-Research-January-March): Productionized Edit Types - https://phabricator.wikimedia.org/T351225#10305517 (10leila) [20:54:06] 06Data-Engineering, 06Product-Analytics, 10Wmfdata-Python: wmfdata should display more progress information and metadata when running a query - https://phabricator.wikimedia.org/T259808#10305537 (10nshahquinn-wmf) The product question here is: what metadata, if any, should be displayed when users run SQL com...