[04:21:31] RECOVERY - Check unit status of monitor_refine_event_sanitized_analytics_immediate on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_event_sanitized_analytics_immediate https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:02:23] (03CR) 10Joal: "Hi @MNeisler, I have questions about the patch before I can comment on the code:" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/736021 (https://phabricator.wikimedia.org/T290516) (owner: 10MNeisler) [07:01:20] 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, and 2 others: Upgrade Superset to 1.3.1 or higher - https://phabricator.wikimedia.org/T288115 (10elukey) @razzi in the Wikitech link there seems to be no final db upgrade done after restoring the superset db dump, did... [08:35:01] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban: Data structuring guidance request - https://phabricator.wikimedia.org/T287402 (10Bumeh-ctr) Thanks for your prompt response @Ottomata. Please @mforns kindly also assist with your expertise inputs. > What do these metric tables look like? Are the sc... [09:56:38] mforns: o/ Did you say that you were able to upgrade me from a Developer to an Admin on this GitLab project? https://gitlab.wikimedia.org/data-engineering/airflow-dags [12:32:30] btullis: got a bit bored today :D have a kerberos automation patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/736753/ [12:33:08] majavah: Brilliant! Many thanks. Looking now. [12:34:10] just a note that I don't really have a way to test it outside what PCC provides, so it may very well be completely broken [12:35:30] Understood. Will bear that in mind, thanks. [12:41:29] (03CR) 10Kosta Harlan: [C: 03+2] Add an image: update schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/736070 (https://phabricator.wikimedia.org/T294659) (owner: 10MewOphaswongse) [12:42:12] (03Merged) 10jenkins-bot: Add an image: update schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/736070 (https://phabricator.wikimedia.org/T294659) (owner: 10MewOphaswongse) [13:02:15] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban: Data structuring guidance request - https://phabricator.wikimedia.org/T287402 (10Ottomata) Just a quick thought without too much more investigation on my part: If the metrics are distinct, then use different tables. If they are the same but with di... [13:23:22] 10Analytics: Data drifts between superset_production on an-coord1001 and db1108 - https://phabricator.wikimedia.org/T279440 (10Ottomata) Am working on {T284150}, and realized that recreating all the 'analytics_meta' databases on db1108 from an-coord100[12] should probably be done first. Then, db1108's analytics... [13:35:28] 10Analytics: Data drifts between superset_production on an-coord1001 and db1108 - https://phabricator.wikimedia.org/T279440 (10elukey) +1 [14:43:36] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10User-razzi: Presto error in Superset - https://phabricator.wikimedia.org/T292879 (10odimitrijevic) [14:46:06] 10Analytics, 10Patch-For-Review: Data drifts between superset_production on an-coord1001 and db1108 - https://phabricator.wikimedia.org/T279440 (10Ottomata) Ok, plan for recreating analytics_meta on db1108. On db1108: ` sudo service mariadb@analytics_meta stop sudo mv /srv/sqldata.analytics_meta /srv/sqldata.... [15:05:58] mforns sukhe are we meeting now? the link for the meeting is not working for me [15:06:38] dsaez: oh? we were meeting today? [15:06:46] nothing on my calendar :( [15:06:48] 10Analytics, 10Patch-For-Review: Data drifts between superset_production on an-coord1001 and db1108 - https://phabricator.wikimedia.org/T279440 (10BTullis) Very minor, but you will probably need a `chown -R mysql:mysql /srv/sqldata.analytics_meta` in there, because the uid/gid values for the `mysql` user are d... [15:07:14] dsaez: I think we had cancelled all meetings for the future, IIRC [15:07:38] I see, somehow is still in my calendar [15:07:58] weird [15:08:03] but makes sense because google meet says that link doesn't exists [15:08:12] yeah I remember we had a weird calendar bug [15:08:44] but yes, no meeting. sorry! but happy to meet otherwise sometime :D [15:18:03] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban: Data structuring guidance request - https://phabricator.wikimedia.org/T287402 (10Bumeh-ctr) Thanks @Ottomata [15:40:23] heya dsaez, I think that meeting is old! I can not see it any more in my calendar, if you can, please feel free to delete it! [15:43:36] 10Analytics, 10Patch-For-Review: Data drifts between superset_production on an-coord1001 and db1108 - https://phabricator.wikimedia.org/T279440 (10elukey) >>! In T279440#7481773, @Ottomata wrote: > Ok, plan for recreating analytics_meta on db1108. > > On db1108: > ` > sudo service mariadb@analytics_meta stop... [15:44:23] sukhe, dsaez: regarding the freezing of the meetings, we data-engineering are focusing on getting Airflow working this quarter and can not dedicate much time to traffic nomalies. However, one of the jobs that we plan to migrate first to Airflow is the anomaly detection jobs (the one that runs traffic anomalies). And when migrated, it will come with a couple improvement I hope, like avoiding repeated alerts, and including the [15:44:24] affected metrics in the email body. That said, if you have something to discuss related to traffic anomalies, please create a one-time meeting! Cheers :] [15:44:43] cool [16:08:23] 10Analytics, 10Data-Engineering, 10Patch-For-Review: Expand Wikiselector to allow more than one wiki - https://phabricator.wikimedia.org/T285050 (10Milimetric) [16:08:34] 10Analytics, 10Data-Engineering: Wikistats should allow more than one project - https://phabricator.wikimedia.org/T283254 (10Milimetric) [16:10:33] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Send some existing Gobblin metrics to prometheus - https://phabricator.wikimedia.org/T294420 (10Milimetric) p:05Triage→03Medium [16:14:35] !log drop and restore superset_staging database to test permissions as they are in production [16:14:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:33:54] question: I am using kerberos::systemd_timer (https://github.com/wikimedia/puppet/blob/production/modules/kerberos/manifests/systemd_timer.pp) in https://github.com/wikimedia/puppet/blob/production/modules/statistics/manifests/product_analytics.pp can I set monitoring_contact_groups to 'analytics-product-users' so if there is an error with execution my team (as members of that posix group) get the alarm? [16:34:52] also I am assuming the alarm is in the form of an email [16:34:57] bearloga: I don't see why not :) ottomata any concern regarding --^ ? [16:39:34] !log add "can sql json on superset" permission to Alpha role on superset.wikimedia.org [16:39:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:41:12] 10Analytics-Radar, 10Data-Engineering, 10Observability-Logging, 10observability, 10Patch-For-Review: Apparent latency warning in 90th centile of eventgate-logging-external - https://phabricator.wikimedia.org/T294911 (10odimitrijevic) [16:42:03] 10Analytics-Radar, 10Observability-Logging, 10observability, 10Patch-For-Review: Apparent latency warning in 90th centile of eventgate-logging-external - https://phabricator.wikimedia.org/T294911 (10Milimetric) [16:51:50] 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, and 2 others: Upgrade Superset to 1.3.1 or higher - https://phabricator.wikimedia.org/T288115 (10razzi) 05In progress→03Resolved @elukey everything is working! I'm not sure what you're referring to with "final db u... [16:55:55] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, and 3 others: wmfdata.mariadb relies on analytics-mysql being available - https://phabricator.wikimedia.org/T292479 (10Milimetric) One idea is to move the logic that finds the host/port to wmfdata, and then to make refinery dep... [16:56:01] 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, and 2 others: Upgrade Superset to 1.3.1 or higher - https://phabricator.wikimedia.org/T288115 (10elukey) I was double checking https://wikitech.wikimedia.org/wiki/User:Razzi/Superset_1.3.1_upgrade_recap and at the end... [16:57:38] 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, and 2 others: Upgrade Superset to 1.3.1 or higher - https://phabricator.wikimedia.org/T288115 (10razzi) Ah yes, @elukey, that was an incomplete paste, I made the embarassing mistake of running a command as superset whe... [16:59:19] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban: Results have expired error in Hue - https://phabricator.wikimedia.org/T294144 (10odimitrijevic) @EYener can you try and run the query through superset? We are not actively maintaining Hue and will be deprecating it eventually. [17:12:34] ottomata: o/ I am working on https://gerrit.wikimedia.org/r/c/operations/puppet/+/736848, is profile::statistics::explorer::ml ok or do you prefer a different name? [17:12:53] I am 99% of the latter so I'll change the name to whatever is best [17:12:54] :D [17:15:46] *99% sure of [17:34:03] +1 luca [17:35:29] no naming battle?? [17:35:31] \o/ [17:35:42] thanks!! [17:54:26] btullis: hi, can I close your finished screen in cumin? [18:01:42] Amir1: I've just done it. Thanks. [18:02:00] Thanks [19:13:57] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics (Kanban): Add SearchSatisfaction to the allowlist - https://phabricator.wikimedia.org/T274607 (10MNeisler) 05Open→03Resolved [19:43:09] 10Analytics-Radar, 10Observability-Alerting, 10Patch-For-Review: Apparent latency warning in 90th centile of eventgate-logging-external - https://phabricator.wikimedia.org/T294911 (10colewhite) [19:43:51] mforns: just saw the message above, thanks! looking forward to it and all the great work you are doing! [19:44:07] Ñ+ [19:44:19] oops, that was a: :] [19:46:49] fancy :D [20:13:56] 10Analytics, 10Patch-For-Review: Data drifts between superset_production on an-coord1001 and db1108 - https://phabricator.wikimedia.org/T279440 (10Ottomata) Thanks both, updated comment with your notes! :) [20:57:18] 10Analytics, 10Data-Engineering, 10Desktop Improvements, 10MediaWiki-extensions-WikimediaEvents, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): Add agent_type and access_method to event data - https://phabricator.wikimedia.org/T294246 (10Jdlrobson) @mforns @Ottomata I was curious about the timeline for ge... [21:01:45] anyone around and want to brainbounce with me [21:01:55] I feel like my brain has bounced with itself long enough, this is getting silly [21:04:10] (03CR) 10Jforrester: Update PHP and version queries (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/730389 (owner: 10Tim Starling) [21:09:06] 10Analytics, 10Data-Engineering, 10Desktop Improvements, 10MediaWiki-extensions-WikimediaEvents, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): Add agent_type and access_method to event data - https://phabricator.wikimedia.org/T294246 (10Ottomata) @odimitrijevic to help prioritize. The code to get this d... [21:11:31] (03CR) 10Cicalese: [C: 03+2] Update PHP and version queries (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/730389 (owner: 10Tim Starling) [21:25:19] (03PS3) 10Bearloga: movement_metrics: Migrate pageviews tables and ETL [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/736583 (https://phabricator.wikimedia.org/T291956) [21:31:33] (03PS4) 10Bearloga: movement_metrics: Migrate pageviews tables and ETL [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/736583 (https://phabricator.wikimedia.org/T291956) [21:37:37] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban: Results have expired error in Hue - https://phabricator.wikimedia.org/T294144 (10EYener) That's great to know @odimitrijevic ! We can work with this data in Superset, yes. Is there a SQL UI that is recommended / supported which we should use going... [21:55:54] (03CR) 10Tim Starling: [C: 03+2] Update PHP and version queries [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/730389 (owner: 10Tim Starling) [22:27:00] (03CR) 10Tim Starling: [V: 03+2 C: 03+2] Update PHP and version queries [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/730389 (owner: 10Tim Starling)