[00:18:15] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9905650 (10Liz) I detected the problem whenever I opened this bug report. I remember filing a different one for Quarry about two weeks ago but this kind of incident, with queries running endlessly and never finishing, only... [00:21:20] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9905654 (10Liz) But hey, a few of my queries just issued reports! I don't know what happened since I posted this message but something has changed for the better. Surprised me. [07:33:55] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9905990 (10Marostegui) [07:34:08] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9905992 (10Marostegui) [07:36:53] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9905995 (10Marostegui) [09:02:37] (03PS18) 10Aqu: Refactor Refine to be triggerd by Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [11:01:53] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9906616 (10Marostegui) [11:51:47] 06Data-Engineering, 06Traffic: Investigate increase in CD termination state after upgrading eqsin/ulsfo to HAProxy 2.8.10 - https://phabricator.wikimedia.org/T367963 (10Fabfur) 03NEW [12:09:47] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9906740 (10KBach) [12:57:25] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9906906 (10kamila) [13:02:31] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9906921 (10kamila) @DMburugu can you please confirm that you have read the [[ https://wikitech.wikimedia.org/wiki/Analytics/Data_access#User_responsibilities... [15:00:17] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9907288 (10DMburugu) Approved [15:02:14] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9907292 (10DMburugu) @Dzahn Sorry for the tag mix up. @kamila Yes, I have read the [[ https://wikitech.wikimedia.org/wiki/Analytics/Data_access#User_responsi... [15:06:23] hi folks! I am rolling out new glibc updates for https://phabricator.wikimedia.org/T367978, nothing really urgent, just a package upgrade [15:06:34] I'll ping you in case something needs a restart [15:24:01] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9907379 (10fnegri) > But hey, a few of my queries just issued reports! I don't know what happened since I posted this message but something has changed for the better. This is likely related to the issues described in T367... [15:31:24] 06Data-Engineering, 10Event-Platform: Event validation errors for mediawiki.page_change.v1 since 2024-03-20 - https://phabricator.wikimedia.org/T367923#9907391 (10Ottomata) Hm. Technically, removing required-ness of a field is a breaking change. jsonschema-tools will require a major version bump. In this c... [15:36:34] (03PS1) 10Ottomata: Make performer field optional in mediawiki/page/change/1.2.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1047549 (https://phabricator.wikimedia.org/T367923) [15:46:43] 06Data-Engineering, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Upgrade hosts to haproxy 2.8.10 - https://phabricator.wikimedia.org/T367756#9907435 (10Fabfur) After upgrading HAProxy to 2.8.10 on whole ulsfo we still see some errors in the kafka DLQ like: ` <134>1 2024-06-19T15:42:31.440224+00... [19:41:08] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9907774 (10Liz) Well, the queries were running and finishing up for a few hours last night (at least it was night where I was at) but now they are back to not completing at all. But for a few hours, everything was in sync. [20:05:02] (03PS1) 10Snwachukwu: Fix MediawikiHistory Checker Null Exceptions [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1047599 (https://phabricator.wikimedia.org/T361016) [21:43:06] 06Data-Engineering, 10Wmfdata-Python: spark.create_custom_session cannot stop a zombie Spark session - https://phabricator.wikimedia.org/T367998 (10nshahquinn-wmf) 03NEW [21:43:44] 06Data-Engineering, 10Wmfdata-Python: Wmfdata: spark session creation fails when there is a zombie Spark session - https://phabricator.wikimedia.org/T367998#9907969 (10nshahquinn-wmf) [21:44:17] 06Data-Engineering, 10Wmfdata-Python: Wmfdata: Spark session creation fails when there is a zombie session - https://phabricator.wikimedia.org/T367998#9907975 (10nshahquinn-wmf)