[02:40:12] (03PS1) 10David Martin: Repair the allowlist for mediawiki_product_metrics_wikifunctions_ui [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114485 (https://phabricator.wikimedia.org/T384531) [03:10:54] (03PS1) 10David Martin: Repair the allowlist for mediawiki_product_metrics_wikilambda_api [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114486 (https://phabricator.wikimedia.org/T384876) [07:54:59] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10499625 (10Marostegui) [07:56:18] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10499626 (10Marostegui) [08:59:07] (03CR) 10Filippo Giunchedi: "Makes sense! Joseph/Gabriele (or anyone else in DE) what do you think of the patch ?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114340 (https://phabricator.wikimedia.org/T383900) (owner: 10Filippo Giunchedi) [09:29:13] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578#10499751 (10Fabfur) [09:48:45] (03CR) 10Gmodena: [C:03+1] "The changes LGTM and should be forward compatibile with the new webrequest haproxykafka feed." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114340 (https://phabricator.wikimedia.org/T383900) (owner: 10Filippo Giunchedi) [12:08:48] (03CR) 10Aqu: [C:03+2] Refine transforms minimalist events support [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1112392 (https://phabricator.wikimedia.org/T383914) (owner: 10Aqu) [12:23:42] (03Merged) 10jenkins-bot: Refine transforms minimalist events support [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1112392 (https://phabricator.wikimedia.org/T383914) (owner: 10Aqu) [12:54:23] (03PS1) 10Aqu: Refine transforms minimalist events support [analytics/refinery/source] (0.2.49) - 10https://gerrit.wikimedia.org/r/1114713 (https://phabricator.wikimedia.org/T383914) [12:56:33] (03PS2) 10Aqu: Refine transforms minimalist events support [analytics/refinery/source] (0.2.49) - 10https://gerrit.wikimedia.org/r/1114713 (https://phabricator.wikimedia.org/T383914) [12:57:39] (03PS3) 10Aqu: Backport: Refine transforms minimalist events support to 0.2.49 [analytics/refinery/source] (0.2.49) - 10https://gerrit.wikimedia.org/r/1114713 (https://phabricator.wikimedia.org/T383914) [13:32:11] (03CR) 10Aqu: [C:03+2] Backport: Refine transforms minimalist events support to 0.2.49 [analytics/refinery/source] (0.2.49) - 10https://gerrit.wikimedia.org/r/1114713 (https://phabricator.wikimedia.org/T383914) (owner: 10Aqu) [13:34:24] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578#10500539 (10Fabfur) [13:42:17] (03Merged) 10jenkins-bot: Backport: Refine transforms minimalist events support to 0.2.49 [analytics/refinery/source] (0.2.49) - 10https://gerrit.wikimedia.org/r/1114713 (https://phabricator.wikimedia.org/T383914) (owner: 10Aqu) [14:07:59] I think I broke stat1010 with an OOM, caused by an ill-judged double left join multiplying tables exponentially. [14:12:28] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Product-Analytics, 10Event-Platform, 13Patch-For-Review: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10500657 (10Ottomata) [14:29:40] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Dumps 2.0 (Kanban Board): Create and test a new produced_by config for the datalake table - https://phabricator.wikimedia.org/T381432#10500733 (10xcollazo) 05In progress→03Resolved >>! In T381432#10493467, @xcollazo wrote: > I can't see the DAG a... [14:35:14] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06Data-Persistence, and 7 others: Add "event_is_test_event" field to "campaign_events" table - https://phabricator.wikimedia.org/T381759#10500758 (10Daimona) 05Resolved→03Open p:05Triage→03Unbreak! It seems like the column exists in so... [14:35:45] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06Data-Persistence, and 7 others: Add "event_is_test_event" field to "campaign_events" table - https://phabricator.wikimedia.org/T381759#10500764 (10Daimona) [14:55:07] (okay, hopefully limited to just my jupyter kernel.) [15:09:57] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Product-Analytics, 10Event-Platform, 13Patch-For-Review: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10500941 (10Ottomata) Had a discussion with @mpopov, @KSarabia-WMF and Ser... [15:18:28] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06Data-Persistence, and 7 others: Add "event_is_test_event" field to "campaign_events" table - https://phabricator.wikimedia.org/T381759#10500961 (10Marostegui) That's vey strange, I'll get this fixed [15:41:48] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06Data-Persistence, and 7 others: Add "event_is_test_event" field to "campaign_events" table - https://phabricator.wikimedia.org/T381759#10501092 (10Marostegui) This should be fixed now. [15:43:21] (03PS1) 10Aqu: Prepare release of 0.2.57 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1114766 [15:46:58] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06Data-Persistence, and 7 others: Add "event_is_test_event" field to "campaign_events" table - https://phabricator.wikimedia.org/T381759#10501100 (10Marostegui) 05Open→03Resolved Confirmed all hosts have the column: ` root@cumin1002:~#... [15:47:03] (03CR) 10Aqu: [V:03+2 C:03+2] "This is just an update of the changelog." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1114766 (owner: 10Aqu) [15:47:59] wikibugs: !log About to deploy analytics/refinery/source 0.2.57 [15:52:33] Starting build #26 for job analytics-refinery-maven-release [16:21:06] Project analytics-refinery-maven-release build #26: 09SUCCESS in 28 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/26/ [16:21:17] Starting build #27 for job analytics-refinery-maven-release [16:39:53] Project analytics-refinery-maven-release build #27: 09SUCCESS in 18 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/27/ [16:43:24] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Product-Analytics, 10Event-Platform, 13Patch-For-Review: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10501310 (10Ottomata) In the interest of time I went ahead with `enrich_fi... [16:43:32] 06Data-Engineering, 10Dumps 2.0: Modify code to dump all slots - https://phabricator.wikimedia.org/T384945 (10xcollazo) 03NEW [17:18:12] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-DomainEvents, 10MW-Interfaces-Team (MWI-Roadmap): DomainEvents - [Hypothesis] WE5.2.6 Event Broadcasting Discovery & Design - https://phabricator.wikimedia.org/T384874#10501449 (10HCoplin-WMF) [17:28:21] Starting build #23 for job analytics-refinery-update-jars [17:28:45] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.57 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114784 [17:28:46] Project analytics-refinery-update-jars build #23: 09SUCCESS in 24 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/23/ [17:58:07] 06Data-Engineering, 06Data-Engineering-Icebox, 10Research-engineering, 06Research-Freezer, 10Event-Platform: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#10501588 (10Isaac) Update: we're moving this to the Research Freezer as we don't expect to work specifically on this in the ne... [19:13:30] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Product-Analytics, 10Event-Platform, 13Patch-For-Review: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10501821 (10Ottomata) FYI, I'm going to try to get done {T383814} at the s... [19:24:11] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10501899 (10Ottomata) [19:24:28] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10501900 (10Ottomata) p:05Triage→03Medium a:03Ottomata [19:24:49] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10501902 (10Ottomata) I've been working on this since my head is in eventgate code as part of {T383814}. I hope to deploy both at the... [19:28:08] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Experimentation Lab, 10Dumps 2.0 (Kanban Board): Implement alerting for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T384962 (10tchin) 03NEW [19:30:11] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Product-Analytics, 10Event-Platform: [Event Platform] Disable default collection of user agent for analytics streams - https://phabricator.wikimedia.org/T384964 (10Ottomata) 03NEW [19:51:44] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Experimentation Lab, 10Dumps 2.0 (Kanban Board): Implement alerting for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T384962#10502020 (10xcollazo) [19:51:47] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Experimentation Lab, 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Dashboard and alerting of data quality metrics for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T357684#10502022 (10xcollazo) [19:58:37] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10502041 (10Ottomata) This is deployed and working in beta. [20:36:50] (03PS1) 10DCausse: Keep event.mediawiki_page_change_v1 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114803 [20:39:25] (03CR) 10DCausse: "I think this table extremely useful, and keeping it might greatly help debugging issues that are discovered long after the retention perio" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114803 (owner: 10DCausse) [21:32:53] !log Running `sudo systemctl restart airflow-scheduler@analytics` to try and clear an SSL error. [21:32:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:45:35] Starting build #24 for job analytics-refinery-update-jars [21:46:00] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.49.3 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114805 [21:46:00] Project analytics-refinery-update-jars build #24: 09SUCCESS in 25 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/24/ [21:53:58] (03PS2) 10Aqu: Add refinery-source jars for v0.2.49.3 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114805 (owner: 10Maven-release-user) [21:54:43] (03CR) 10Aqu: [C:03+2] Add refinery-source jars for v0.2.57 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114784 (owner: 10Maven-release-user) [21:54:47] (03CR) 10Aqu: [V:03+2 C:03+2] Add refinery-source jars for v0.2.57 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114784 (owner: 10Maven-release-user) [21:55:05] (03CR) 10Aqu: [C:03+2] Add refinery-source jars for v0.2.49.3 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114805 (owner: 10Maven-release-user) [21:55:09] (03CR) 10Aqu: [V:03+2 C:03+2] Add refinery-source jars for v0.2.49.3 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114805 (owner: 10Maven-release-user) [21:56:29] !log Deployed refinery-source using jenkins [21:56:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:56:50] !log About to deploy analytics/refinery [21:56:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:09:01] !log Deployed refinery-source using jenkins [22:37:00] (03CR) 10Ottomata: [C:03+1] Keep event.mediawiki_page_change_v1 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114803 (owner: 10DCausse) [23:13:49] 06Data-Engineering: Implement Blunderbuss HDFS synchronization for airflow-dags repository - https://phabricator.wikimedia.org/T384984#10502638 (10amastilovic)