[02:19:04] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [02:19:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=codfw.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [02:48:26] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9901968 (10Liz) The status says this case is open and is a high priority but it's not getting any attention from those who might be in a position to resolve this problem. I've been told that "complaining" doesn't help bu... [03:19:04] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [03:19:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=codfw.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [04:20:25] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#9901992 (10Marostegui) a:03ABran-WMF [05:41:12] 06Data-Engineering: Publishing conda environments with WMF Data Workflow Utils is broken - https://phabricator.wikimedia.org/T367848 (10KCVelaga_WMF) 03NEW [05:44:31] 06Data-Engineering: Publishing conda environments with WMF Data Workflow Utils is broken - https://phabricator.wikimedia.org/T367848#9902098 (10KCVelaga_WMF) p:05Triage→03High This is blocking at least three tasks for me: T362615, T367016 & T366869#9891978 (all of which need to access MariaDB from Airflow). [07:37:48] Seems like the Quarry exclude change either wasn't deployed or did nothing [07:42:09] !log update miniconda version on `an-test-client` T356231 [07:42:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:42:12] T356231: Package versions in Conda-Analytics are not pinned - https://phabricator.wikimedia.org/T356231 [08:34:03] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9902324 (10fnegri) 05Open→03In progress [08:35:26] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9902321 (10fnegri) @Liz it is getting attention by multiple people, but it's not clear what the problem is. :) It might also be related to some performance issues that started last Friday on one database (T367778). This is... [08:37:21] 06Data-Engineering, 10Data-Platform-SRE (2024.06.17 - 2024.07.07): Publishing conda environments with WMF Data Workflow Utils is broken - https://phabricator.wikimedia.org/T367848#9902340 (10Stevemunene) a:03Stevemunene Hello, Adding DPE SRE to the ticket as well. We encountered a similar error on https://p... [08:48:31] 06Data-Engineering, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 10Event-Platform: mw-page-content-change-enrich flink app is missing in k8s staging - https://phabricator.wikimedia.org/T367116#9902390 (10Gehel) [09:11:23] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856 (10Zabe) 03NEW [09:11:46] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9902490 (10Zabe) p:05Triage→03Low (more like lowest) [09:12:28] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9902493 (10Marostegui) a:03Marostegui [09:38:18] 06Data-Engineering, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 10Event-Platform: mw-page-content-change-enrich flink app is missing in k8s staging - https://phabricator.wikimedia.org/T367116#9902613 (10gmodena) @amastilovic @Ottomata Can we close this task? Pods have been up and running for a week, with... [09:53:04] 06Data-Engineering, 10ChangeProp, 10observability, 10service-runner, 10Event-Platform: Upgrade prom-client in NodeJS service-runner and enable collectDefaultMetrics - https://phabricator.wikimedia.org/T350180#9902634 (10gmodena) Apologies for the lack of activity on this task, it somehow fell through the... [10:05:10] 06Data-Engineering, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Upgrade hosts to haproxy 2.8.10 - https://phabricator.wikimedia.org/T367756#9902659 (10Fabfur) [10:13:19] 06Data-Engineering, 06Data Products, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9902686 (10Bugreporter) [11:40:08] 06Data-Engineering, 06Data-Platform-SRE: [Iceberg Migration] P.O.C. on Iceberg sensor using Postgres table to keep status of updates - https://phabricator.wikimedia.org/T340466#9902941 (10lbowmaker) 05Open→03Declined Declining. We will not implement this approach and will likely favor a combined use of... [12:35:57] 06Data-Engineering, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 10Event-Platform: mw-page-content-change-enrich flink app is missing in k8s staging - https://phabricator.wikimedia.org/T367116#9903092 (10Ottomata) Ya let's close. Perhaps, we can add a pointer in docs somewhere to this issue. I had forg... [12:45:21] 14Analytics, 06Data-Engineering-Icebox: EventGate throttling and DOS prevention - https://phabricator.wikimedia.org/T256891#9903149 (10Ottomata) [12:45:21] 14Analytics, 14Analytics-Kanban, 10MediaWiki-extensions-EventLogging, 10Event-Platform: Modern Event Platform: Stream Intake Service (EventGate): Implementation - https://phabricator.wikimedia.org/T206785#9903150 (10Ottomata) [12:45:59] 14Analytics, 06Data-Engineering-Icebox: EventGate throttling and DOS prevention - https://phabricator.wikimedia.org/T256891#9903152 (10Ottomata) Related: {T306580} [13:14:19] 10Data-Engineering (Q4 2024 April 1st - June 30th), 13Patch-For-Review: [Refine refactoring] Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9903232 (10Ottomata) [13:14:20] 06Data-Engineering, 10Event-Platform: Event Platform schemas should not support type changes to structs as array element or map value types - https://phabricator.wikimedia.org/T366487#9903233 (10Ottomata) [13:16:38] 10Data-Engineering (Q4 2024 April 1st - June 30th), 13Patch-For-Review: [Refine refactoring] Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9903236 (10Ottomata) Hi, we probably do {T366487} along with this work. It is not urgent (it has been the status quo fo... [13:49:03] 06Data-Engineering: [Iceberg Migration] P.O.C. on Iceberg sensor using Iceberg table to keep status of updates - https://phabricator.wikimedia.org/T340463#9903396 (10lbowmaker) 05Open→03Declined Declining. We will not implement this approach and will likely favor a combined use of Airflows ExternalTaskSe... [13:49:10] 06Data-Engineering: [Iceberg Migration] P.O.C. on Iceberg sensor using Snapshot metadata to keep status of updates - https://phabricator.wikimedia.org/T340471#9903402 (10lbowmaker) 05Open→03Declined Declining. We will not implement this approach and will likely favor a combined use of Airflows ExternalTa... [14:23:45] !log commencing roll-reboot of an-presto workers for T366555 [14:23:47] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:23:47] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9903568 (10diego) Thanks @JAllemandou ! [14:23:56] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9903570 (10diego) 05Open→03Resolved [14:51:13] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9903671 (10Wurgl) @fnegri: My query had this problem since June 6th. [15:02:18] !log deployed airflow analytics to update CIM category allow-list [15:02:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:14:06] 06Data-Engineering, 10Data Pipelines, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Upgrade Airflow to 2.9.2 - https://phabricator.wikimedia.org/T365449#9903783 (10Gehel) [15:46:26] 06Data-Engineering, 10CirrusSearch, 03Discovery-Search (Current work): [Search Update Pipeline] Source streams for private wikis - https://phabricator.wikimedia.org/T346046#9903892 (10Ottomata) Discussed this in a meeting with Gabriele today. We discussed an ideal solution long term solution, and also a pra... [15:55:06] 06Data-Engineering, 10CirrusSearch, 03Discovery-Search (Current work): [Search Update Pipeline] Source streams for private wikis - https://phabricator.wikimedia.org/T346046#9903943 (10Ottomata) As I wrote the 'Practical Short Term Solution' I came up again against the awkwardness of the `wgEnableEventBus` co... [16:49:40] 10Quarry: github action building main rather than branch - https://phabricator.wikimedia.org/T367630#9904225 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/58 [16:51:50] 06Data-Engineering, 06Tech-Docs-Team, 05Goal: Redesign Data Platform docs on Wikitech - https://phabricator.wikimedia.org/T350911#9904235 (10TBurmeister) Continued cleanup after the big doc migration: - Created a [[ https://wikitech.wikimedia.org/wiki/Data_Platform/AQS | new landing page for AQS docs ]], to... [16:52:51] 10Quarry: github action building main rather than branch - https://phabricator.wikimedia.org/T367630#9904248 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/58 [17:00:02] 10Quarry: github action building main rather than branch - https://phabricator.wikimedia.org/T367630#9904269 (10rook) 05Open→03Resolved a:03rook [17:04:56] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9904291 (10fnegri) > If you look through Execution time column on Recent queries list, it actually seems like that results of virtually any query with execution time longer than ~120s will never make it back I had to scrol... [20:37:59] 06Data-Engineering, 10Event-Platform: Event validation errors for mediawiki.page_change.v1 since 2024-03-20 - https://phabricator.wikimedia.org/T367923 (10Ottomata) 03NEW [20:47:18] 06Data-Engineering, 10Event-Platform: Event validation errors for mediawiki.page_change.v1 since 2024-03-20 - https://phabricator.wikimedia.org/T367923#9905133 (10Ottomata) Hm. I think I introduced this bug in {T342487}. We are not setting `performer` correctly, but we never made performer a non-required f... [21:05:58] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9905205 (10Dzahn) [21:12:31] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9905218 (10Dzahn) Hi @Kgraessle in addition to your manager please get any of the following people to approve of this request here on the ticket. ` a... [21:12:32] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9905219 (10Dzahn) [21:20:12] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9905262 (10Dzahn) [21:21:14] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9905264 (10Dzahn) tagging with data-engineering per the new process to request approval from group approvers [21:21:19] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9905266 (10Dzahn) it's an SRE access request, unrelated to LDAP. adjusting tags [21:21:32] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9905267 (10Dzahn)