[01:15:34] PROBLEM - Webrequests Varnishkafka log producer on cp5029 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [01:35:36] RECOVERY - Webrequests Varnishkafka log producer on cp5029 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [09:17:04] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10321641 (10tchin) Just a note that if we want to do DQ with PyDeeq... [10:08:22] (03PS3) 10Joal: Update time-partitioned-flag data publisher [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1078623 (https://phabricator.wikimedia.org/T376144) [10:09:14] (03CR) 10CI reject: [V:04-1] Update time-partitioned-flag data publisher [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1078623 (https://phabricator.wikimedia.org/T376144) (owner: 10Joal) [10:09:16] (03CR) 10Joal: "I have updated the patch to only have one heuristic, the low-volume one" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1078623 (https://phabricator.wikimedia.org/T376144) (owner: 10Joal) [10:25:47] (03PS4) 10Joal: Update time-partitioned-flag data publisher [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1078623 (https://phabricator.wikimedia.org/T376144) [10:41:21] (03PS3) 10Milimetric: Include user_is_temp column in user table sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1087964 (https://phabricator.wikimedia.org/T379631) [13:24:30] (03CR) 10Milimetric: "Testing with (in screens, with `sudo -u analytics /bin/bash`):" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1087964 (https://phabricator.wikimedia.org/T379631) (owner: 10Milimetric) [14:10:09] 10Data-Engineering (Q2 2024 October 1st - December 31th), 13Patch-For-Review: [HAProxy transition] Deploy a staging airflow dag for webrequest refinement - https://phabricator.wikimedia.org/T378342#10322573 (10gmodena) >>! In T378342#10267687, @CodeReviewBot wrote: > gmodena opened https://gitlab.wikimedia.org... [16:38:37] 06Data-Engineering, 06MW-Interfaces-Team, 10Event-Platform: Explore a mechanism for publishing domain events to an event bus - https://phabricator.wikimedia.org/T379935 (10daniel) 03NEW [16:45:55] 06Data-Engineering, 06MW-Interfaces-Team, 10Event-Platform: Make DomainEvents serializable - https://phabricator.wikimedia.org/T379936 (10daniel) 03NEW [16:57:52] 06Data-Engineering, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, 10Observability-Metrics, and 4 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10323305 (10lmata) [16:58:15] 06Data-Engineering, 06MW-Interfaces-Team, 10Event-Platform: Explore a mechanism for publishing domain events to an event bus - https://phabricator.wikimedia.org/T379935#10323299 (10Ottomata) This was originally posted on {T379775}, but is more relevant here. --- Okay, I've written up some thoughts on how an... [18:50:57] 06Data-Engineering, 06MW-Interfaces-Team, 10Event-Platform: Explore a mechanism for publishing domain events to an event bus - https://phabricator.wikimedia.org/T379935#10323852 (10Ottomata) Oh, is this a duplicate of {T378933} ? [19:03:03] (03CR) 10Mforns: Update MediaWiki History to support Temp Accounts (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1088232 (https://phabricator.wikimedia.org/T379230) (owner: 10Mforns) [19:09:37] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, and 2 others: Design and document new Domain Events feature in MediaWiki core - https://phabricator.wikimedia.org/T379959 (10Ottomata) 03NEW [19:11:13] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, and 2 others: Design and document new Domain Events feature in MediaWiki core - https://phabricator.wikimedia.org/T379959#10323976 (10Ottomata) I created this because I wanted a visible and semi-centralized [[ https://wikit... [19:12:34] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, and 2 others: Design and document new Domain Events feature in MediaWiki core - https://phabricator.wikimedia.org/T379959#10323981 (10Ottomata) [19:14:54] (03CR) 10Mforns: Add New Edit Hourly HQL for Temp Account Change (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767) (owner: 10Jennifer Ebe) [19:21:31] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, and 2 others: Design and document new Domain Events feature in MediaWiki core - https://phabricator.wikimedia.org/T379959#10323997 (10Ottomata) @daniel should we consider https://event.thephpleague.com/3.0/ ? in your compar... [20:00:25] (03CR) 10Gehel: Extraction of RefineHelper (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1080706 (owner: 10Gehel) [20:03:45] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, and 2 others: Design and document new Domain Events feature in MediaWiki core - https://phabricator.wikimedia.org/T379959#10324203 (10Ottomata) Moving a conversation from the design doc here. === Sinks and Sources terminol... [20:49:26] 06Data-Engineering, 10MediaWiki-Core-Hooks, 06MW-Interfaces-Team, 10Event-Platform, and 2 others: Design and document new Domain Events feature in MediaWiki core - https://phabricator.wikimedia.org/T379959#10324344 (10Ottomata) === `$eventType` vs `$channel` In the current implementation, listeners are r...