[07:26:49] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1073214 (https://phabricator.wikimedia.org/T368788) (owner: 10Joal) [07:28:09] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1075075 (https://phabricator.wikimedia.org/T375433) (owner: 10Gerrit maintenance bot) [07:28:44] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1075073 (https://phabricator.wikimedia.org/T375424) (owner: 10Gerrit maintenance bot) [07:38:38] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to stat1007 for cyndywikime - https://phabricator.wikimedia.org/T375060#10170093 (10Vgutierrez) 05Stalled→03In progress [07:39:34] 10Data-Engineering (Q1 2024 July 1st - September 30th): Handle Late-Arrived Events from Gobblin into Airflow triggered Refine - https://phabricator.wikimedia.org/T370665#10170099 (10JAllemandou) After explaining my finding to the team yesterday, here are the following steps: I'm gonna have a look at late-events... [07:50:19] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to stat1007 for cyndywikime - https://phabricator.wikimedia.org/T375060#10170102 (10Vgutierrez) 05In progress→03Resolved ` vgutierrez@krb1001:~$ sudo manage_principals.py create cyndywikime --email_address=csimi... [08:20:26] (03Restored) 10DCausse: Add wikibase/rdf/update_stream/1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/594098 (owner: 10DCausse) [09:00:08] (03PS1) 10Aqu: Fix parse_user_agent following Refine refactoring [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1075145 [09:10:26] (03PS2) 10Aqu: Fix parse_user_agent following Refine refactoring [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1075145 (https://phabricator.wikimedia.org/T369845) [09:11:34] (03PS3) 10Aqu: Fix parse_user_agent following Refine refactoring [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1075145 (https://phabricator.wikimedia.org/T369845) [09:13:19] (03PS4) 10Aqu: Fix parse_user_agent following Refine refactoring [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1075145 (https://phabricator.wikimedia.org/T369845) [10:41:13] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Data Quality] [SPIKE] Can we identify indicators to inform an SLO for event emission and intake? - https://phabricator.wikimedia.org/T345195#10170635 (10gmodena) F/up from a conversation we had at sync. >>... [12:13:41] 06Data-Engineering, 06Data-Platform-SRE, 06serviceops, 06SRE, 10Event-Platform: DRY kafka broker declaration in helmfiles - https://phabricator.wikimedia.org/T253058#10170935 (10JMeybohm) [13:10:32] !log uncordoned dse-k8s-worker1001 and draining dse-k8s-worker1002 ready for reimage for T365283 [13:10:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:10:36] T365283: Reimage dse-k8s-worker100[1-4] to correct the partman recipe and enable additional local storage - https://phabricator.wikimedia.org/T365283 [16:01:24] 06Data-Engineering, 10Data-Platform-SRE (2024.09.06 - 2024.09.27): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10172189 (10brouberol) Additional thought: once we migrate to `KubernetesExecutor` instead of `LocalExecutor`, the dags repo would get cloned by `git-sync` at t... [16:12:41] !log Deploying Refinery [16:12:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:51:05] !log Manually rerun refinery-import-mediawiki-page-dumps to take inti account not importing labswiki [16:51:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:54:03] !log Deployed refinery using scap, then deployed onto hdfs [16:54:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:57:29] 06Data-Engineering, 10Data Products (Data Products Sprint 19): Add wikitech (labswiki) to the sqoop list - https://phabricator.wikimedia.org/T217792#10172352 (10BTullis) Now that `labswiki` is a normal database, on the s6 section, are we able to start dumping it, as per all of the other databases? It looks lik... [17:18:17] 10Quarry: Quarry login fails due to redirect to plaintext HTTP URL - https://phabricator.wikimedia.org/T361471#10172447 (10LucasWerkmeister) Still happening. I got redirected like this: - https://quarry.wmcloud.org/login?next=/ - https://meta.wikimedia.org/w/index.php?title=Special%3AOAuth%2Fauthenticate&oauth_t... [17:28:14] 10Quarry: Quarry login fails due to redirect to plaintext HTTP URL - https://phabricator.wikimedia.org/T361471#10172471 (10github-toolforge-bot) supertassu opened https://github.com/toolforge/quarry/pull/70 [17:33:45] (03CR) 10DCausse: "moved to https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/merge_requests/2" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/594098 (owner: 10DCausse) [17:35:15] (03Abandoned) 10DCausse: Add wikibase/rdf/update_stream/1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/594098 (owner: 10DCausse) [17:44:53] (03Restored) 10DCausse: Add wikibase/rdf/update_stream/1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/594098 (owner: 10DCausse) [17:45:52] (03PS4) 10DCausse: Add /mediawiki/wikibase/entity/rdf_change/2.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/594098 (https://phabricator.wikimedia.org/T374918) [17:48:40] (03PS5) 10DCausse: Add /mediawiki/wikibase/entity/rdf_change/2.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/594098 (https://phabricator.wikimedia.org/T374918) [19:15:45] (03PS6) 10DCausse: Add /mediawiki/wikibase/entity/rdf_change/2.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/594098 (https://phabricator.wikimedia.org/T374918) [20:17:46] 06Data-Engineering, 03Discovery-Search (Current work): Datahub - ingest Hive discovery database - https://phabricator.wikimedia.org/T374118#10172998 (10EBernhardson) Looking for tables that contain the column "source_text"only finds the table for the update pipeline event stream, but not the cirrus index dumps... [20:42:26] 06Data-Engineering: [SPIKE] Learn and document how to use Flink-CDC from MediaWiki MariaDB locally - https://phabricator.wikimedia.org/T373144#10173047 (10NoZeroDay) @Ottomata Thanks for bringing this to my attention, Andrew. [23:07:25] (03PS1) 10Máté Szabó: ipinfo_interaction: Add event_ip_data_source [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1075329 (https://phabricator.wikimedia.org/T356105) [23:07:58] (03CR) 10CI reject: [V:04-1] ipinfo_interaction: Add event_ip_data_source [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1075329 (https://phabricator.wikimedia.org/T356105) (owner: 10Máté Szabó) [23:10:02] (03PS2) 10Máté Szabó: ipinfo_interaction: Add event_ip_data_source [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1075329 (https://phabricator.wikimedia.org/T356105) [23:10:29] (03CR) 10CI reject: [V:04-1] ipinfo_interaction: Add event_ip_data_source [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1075329 (https://phabricator.wikimedia.org/T356105) (owner: 10Máté Szabó) [23:12:44] (03PS3) 10Máté Szabó: ipinfo_interaction: Add event_ip_data_source [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1075329 (https://phabricator.wikimedia.org/T356105)