[12:32:57] 06Data-Engineering, 10Data-Engineering-Wikistats, 10Data Pipelines, 06Data Products, and 3 others: Merge ks-Arab and ks-Deva to ks - https://phabricator.wikimedia.org/T314476#9930309 (10MaryMunyoki) [15:30:21] 10Data-Engineering (Q4 2024 April 1st - June 30th), 13Patch-For-Review: [Refine refactoring] Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9931089 (10Antoine_Quhen) To check the output of the new Refine process, I've been working with diff from https://githu... [15:45:59] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Refine Refactoring] Integrate Refine workflow configuration into ESC - https://phabricator.wikimedia.org/T367134#9931166 (10Antoine_Quhen) As discussed in the meeting, I'm OK with minimal changes to ESC: It could be nice to add the hourly_computing_scale and h... [15:48:39] 10Data-Engineering (Q4 2024 April 1st - June 30th), 13Patch-For-Review: [Refine refactoring] Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9931172 (10Ottomata) > The user_agent structure depends on the existing table schema Yeah, this is a legacy EventLoggin... [15:51:14] 06Data-Engineering, 06Movement-Insights, 06Research-Freezer: Investigate relation of Prefetch feature to increase in automated traffic and impact on unique devices - https://phabricator.wikimedia.org/T336715#9931178 (10OSefu-WMF) 05Open→03Resolved Per T364126 we made the decision to turn off Chrome p... [16:01:55] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Refine Refactoring] Integrate Refine workflow configuration into ESC - https://phabricator.wikimedia.org/T367134#9931250 (10Ottomata) > have a boolean ~needs_to_be_refined. This would be a consumer specific configuration in ESC. Gobblin currently uses the co... [16:42:56] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9931405 (10Dzahn) [16:45:28] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9931412 (10Dzahn) 05In progress→03Resolved You have now been added to the group as requested. Feel free to try the Superset acc... [17:04:21] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9931522 (10Dzahn) 05In progress→03Resolved You have now been added to the group as requested. Feel free to test the Superset a... [17:09:40] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9931555 (10Dzahn) [17:10:58] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9931539 (10Dzahn) 05In progress→03Resolved You have now been added to the group as requested. Feel free to test the ne... [17:48:30] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Refine Refactoring] Integrate Refine workflow configuration into ESC - https://phabricator.wikimedia.org/T367134#9931756 (10Ottomata) We discussed a bunch in a meeting today. We have two main ideas on how to structure ESC for Airflow Refine. First, some su... [19:39:58] 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Degraded RAID on aqs1013 - https://phabricator.wikimedia.org/T368564#9932375 (10Dzahn) [19:40:00] 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Degraded RAID on aqs1013 - https://phabricator.wikimedia.org/T368564#9932380 (10Dzahn) [19:41:07] 06Data-Engineering, 10Event-Platform: [Event Platform] mw-page-content-change-enrich down in eqiad 2024-06-27 - https://phabricator.wikimedia.org/T368667 (10Ottomata) 03NEW [20:02:33] 06Data-Engineering, 10Event-Platform: [Event Platform] mw-page-content-change-enrich down in eqiad 2024-06-27 - https://phabricator.wikimedia.org/T368667#9932472 (10Ottomata) Deleting JobManager did not work. @bking tried to delete the app and re deploy. Same error both times: `lang=json {"@timestamp":"2024... [20:10:34] 06Data-Engineering, 10Event-Platform: [Event Platform] mw-page-content-change-enrich down in eqiad 2024-06-27 - https://phabricator.wikimedia.org/T368667#9932482 (10Ottomata) @bking deleted and redeployed the app. It started up and I think I got the root cause: `lang=json { "@timestamp":"2024-06-27T20:0... [20:17:33] 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install an-conf100[4-6] - https://phabricator.wikimedia.org/T364429#9932489 (10Jclark-ctr) @BTullis can you update preseed.yam and site.pp file for these servers [20:20:59] 06Data-Engineering, 10Event-Platform: [Event Platform] mw-page-content-change-enrich down in eqiad 2024-06-27 - https://phabricator.wikimedia.org/T368667#9932495 (10Ottomata) AH! I know why! {T367923} I did not deploy https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1047995 and the MW train we... [20:31:30] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.11; 2024-06-25): Event validation errors for mediawiki.page_change.v1 due to missing performer field on revision suppressions - https://phabricator.wikimedia.org/T367923#9932527 (10Ottomata) > I'll redeploy the... [20:31:37] 06Data-Engineering, 10Event-Platform: [Event Platform] mw-page-content-change-enrich down in eqiad 2024-06-27 - https://phabricator.wikimedia.org/T368667#9932529 (10Ottomata) Summary here: https://phabricator.wikimedia.org/T367923#9932527 [20:48:20] 06Data-Engineering, 10Event-Platform: [Event Platform] mw-page-content-change-enrich down in eqiad 2024-06-27 - https://phabricator.wikimedia.org/T368667#9932586 (10Ottomata) 05Open→03Resolved a:03Ottomata Should be fixed. I'm going to resolve this ticket. Let's do any follow up discussions in T367... [20:49:41] FIRING: MediawikiPageContentChangeEnrichHighKafkaConsumerLag: ... [20:49:41] High Kafka consumer lag for mw_page_content_change_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichHighKafkaConsumerLag [20:51:41] 06Data-Engineering, 10Event-Platform: [Event Platform] mw-page-content-change-enrich down in eqiad 2024-06-27 - https://phabricator.wikimedia.org/T368667#9932606 (10gmodena) @Ottomata ack. Alert manager might fire some alerts about kafka consumer lag (it's one of our SLIs). That's expected while the app ba... [20:57:39] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.11; 2024-06-25), 13Patch-For-Review: Event validation errors for mediawiki.page_change.v1 due to missing performer field on revision suppressions - https://phabricator.wikimedia.org/T367923#9932628 (10xcollazo... [21:03:20] 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install an-conf100[4-6] - https://phabricator.wikimedia.org/T364429#9932633 (10VRiley-WMF) [21:14:41] RESOLVED: MediawikiPageContentChangeEnrichHighKafkaConsumerLag: ... [21:14:41] High Kafka consumer lag for mw_page_content_change_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichHighKafkaConsumerLag [21:27:24] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.11; 2024-06-25), 13Patch-For-Review: Event validation errors for mediawiki.page_change.v1 due to missing performer field on revision suppressions - https://phabricator.wikimedia.org/T367923#9932681 (10Ottomata... [23:01:31] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar): Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9932970 (10Krinkle) This redirect is coming from Apache and known as `Director...