[02:38:35] PROBLEM - Webrequests Varnishkafka log producer on cp5032 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [02:38:35] PROBLEM - Webrequests Varnishkafka log producer on cp5031 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [02:38:36] PROBLEM - Webrequests Varnishkafka log producer on cp5028 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [02:39:35] RECOVERY - Webrequests Varnishkafka log producer on cp5032 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [02:42:35] RECOVERY - Webrequests Varnishkafka log producer on cp5031 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [02:46:35] PROBLEM - Webrequests Varnishkafka log producer on cp5025 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [02:51:35] RECOVERY - Webrequests Varnishkafka log producer on cp5028 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [03:11:35] RECOVERY - Webrequests Varnishkafka log producer on cp5025 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [06:23:00] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#10224609 (10ABran-WMF) [07:20:24] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data-Platform-SRE, 06Movement-Insights: 2024-10-10 Data Loss Incident - webrequest Hive table - https://phabricator.wikimedia.org/T376882#10224647 (10JAllemandou) Thanks a lot for fixing the data-deletion checksum @xcollazo ! [07:21:17] (03CR) 10Joal: [V:03+2 C:03+2] "Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1079672 (https://phabricator.wikimedia.org/T377088) (owner: 10Gerrit maintenance bot) [07:43:51] 14Data-Engineering (Sprint 5), 06Data-Platform-SRE: Globally configure spark to use fileoutputcommitter.algorithm.version=2 to avoid concurrent write issues - https://phabricator.wikimedia.org/T351388#10224692 (10JAllemandou) >>! In T351388#10221363, @Ottomata wrote: >> actually we should have set the paramete... [08:15:10] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform, 13Patch-For-Review: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10224756 (10pfischer) The batch-only sink is ready. Two questions remain open: * Naming: What should we name this sink?... [09:11:11] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform, 13Patch-For-Review: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10224898 (10dcausse) >>! In T374341#10224756, @pfischer wrote: > * Early vs late schema validation: Currently the validati... [10:49:48] 14Data-Engineering (Sprint 5), 06Data-Platform-SRE, 13Patch-For-Review: Globally configure spark to use fileoutputcommitter.algorithm.version=2 to avoid concurrent write issues - https://phabricator.wikimedia.org/T351388#10225323 (10BTullis) I have restored this patch, which had previously been abandoned: ht... [11:44:07] !log restarted postgresql@13-main on an-db1001, followed by all airflow schedulers, for T374240 [11:44:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:45:43] hello! o/ I'm just rolling out a change to remove the unused aqsv1 support in restbase atm [11:46:07] and if that goes okay I'll be removing the aqs1 nodejs service https://gerrit.wikimedia.org/r/c/operations/puppet/+/1075163 [11:46:10] any objections? [11:54:30] hnowlan: No objections from me. [11:55:03] !log roll-restarting nginx and envoy on wcqs-public nodes for T374240 [11:55:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:55:20] thanks! [12:31:27] 06Data-Engineering, 06Discovery-Search, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.20 - https://phabricator.wikimedia.org/T376812#10225610 (10gmodena) F/up from a conversation in slack with @dcausse. Last Friday, I time-boxed some grooming. What I found so far is that `flink-connector-ka... [12:50:47] 06Data-Engineering, 06Discovery-Search: Bump eventutilities to 1.19 - https://phabricator.wikimedia.org/T377130 (10dcausse) 03NEW [12:50:53] 06Data-Engineering, 06Discovery-Search: Bump eventutilities to 1.19 - https://phabricator.wikimedia.org/T377130#10225701 (10dcausse) Current attempt by @gmodena uploaded at https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/1079506 But failing with ` [ERROR] Errors: [ERROR] TestEventRowSerializer.... [12:54:19] 06Data-Engineering, 06Discovery-Search, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.20 - https://phabricator.wikimedia.org/T376812#10225724 (10dcausse) >>! In T376812#10225610, @gmodena wrote: > @dcausse I'm not sure how you'd like to organize the work, but perhaps we can create a dedicate... [12:54:49] 06Data-Engineering, 06Discovery-Search: Bump eventutilities to support flink 1.19 - https://phabricator.wikimedia.org/T377130#10225729 (10dcausse) [12:56:44] 06Data-Engineering, 06Discovery-Search: Bump eventutilities to support flink 1.19 - https://phabricator.wikimedia.org/T377130#10225743 (10dcausse) [12:58:14] 06Data-Engineering, 06Discovery-Search, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.19 - https://phabricator.wikimedia.org/T376812#10225751 (10dcausse) [13:02:12] 06Data-Engineering, 06Discovery-Search: Create and distribute a flink base image with flink 1.19.1 - https://phabricator.wikimedia.org/T377134 (10dcausse) 03NEW [13:22:18] 06Data-Engineering, 06Discovery-Search: Upload an image with flink-k8s-operator version that supports flink 1.19 - https://phabricator.wikimedia.org/T377137 (10dcausse) 03NEW [13:23:00] 06Data-Engineering, 06Discovery-Search, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.19 - https://phabricator.wikimedia.org/T376812#10225896 (10dcausse) [13:34:23] 06Data-Engineering, 10Dumps 2.0, 10Event-Platform, 13Patch-For-Review: [SPIKE] how can we support Spark producer/consumers in Event Platform - https://phabricator.wikimedia.org/T374341#10225932 (10pfischer) @dcausse, according to @JAllemandou, the validation-related code lives in [[ https://github.com/wiki... [13:54:04] 06Data-Engineering, 06Discovery-Search, 13Patch-For-Review: Bump eventutilities to support flink 1.19 - https://phabricator.wikimedia.org/T377130#10226019 (10gmodena) >>! In T377130#10225691, @dcausse wrote: > Current attempt by @gmodena uploaded at https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/... [15:18:52] 06Data-Engineering, 06Discovery-Search, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.19 - https://phabricator.wikimedia.org/T376812#10226406 (10Gehel) p:05Triage→03Medium [15:19:03] 06Data-Engineering, 06Discovery-Search, 13Patch-For-Review: Bump eventutilities to support flink 1.19 - https://phabricator.wikimedia.org/T377130#10226408 (10Gehel) p:05Triage→03Medium [15:19:16] 06Data-Engineering, 06Discovery-Search: Upload an image with flink-k8s-operator version that supports flink 1.19 - https://phabricator.wikimedia.org/T377137#10226412 (10Gehel) p:05Triage→03Medium [15:19:32] 06Data-Engineering, 06Discovery-Search: Create and distribute a flink base image with flink 1.19.1 - https://phabricator.wikimedia.org/T377134#10226410 (10Gehel) p:05Triage→03Medium [15:20:52] 06Data-Engineering, 03Discovery-Search (Current work), 13Patch-For-Review: Bump eventutilities to support flink 1.19 - https://phabricator.wikimedia.org/T377130#10226414 (10Gehel) [15:21:53] 06Data-Engineering, 03Discovery-Search (Current work), 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.19 - https://phabricator.wikimedia.org/T376812#10226417 (10Gehel) [15:22:09] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search: Upload an image with flink-k8s-operator version that supports flink 1.19 - https://phabricator.wikimedia.org/T377137#10226418 (10Gehel) [15:23:03] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search: Create and distribute a flink base image with flink 1.19.1 - https://phabricator.wikimedia.org/T377134#10226422 (10Gehel) [15:26:03] 06Data-Engineering, 03Discovery-Search (Current work), 07Epic, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.19 - https://phabricator.wikimedia.org/T376812#10226446 (10Gehel) [15:30:55] 06Data-Engineering, 10Wikidata, 03Discovery-Search (Current work), 10Event-Platform: Configure https://stream.wikimedia.org to expose rdf-streaming-updater.mutation - https://phabricator.wikimedia.org/T374921#10226456 (10Gehel) [17:11:38] 14Analytics-Radar, 06Data-Engineering-Icebox, 10MediaWiki-Page-editing, 10Two-Column-Edit-Conflict-Merge, 10research-ideas: statistics about edit conflicts according to page type - https://phabricator.wikimedia.org/T139019#10226841 (10Pppery) [21:51:37] 06Data-Engineering, 06Data-Platform, 06DBA, 07Schema-change-in-production: Change page.page_links_updated to fixed-length timestamp in wmf wikis - https://phabricator.wikimedia.org/T371742#10227278 (10Ladsgroup)