[00:28:18] (DruidSegmentsUnavailable) firing: More than 30 segments have been unavailable for wmf_netflow on the druid_analytics Druid cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid/Alerts#Druid_Segments_Unavailable - https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&var-cluster=druid_analytics&panelId=49&fullscreen&orgId=1&var-cluster=druid_analytics - https://alerts.wikimedia.org [00:28:18] (DruidSegmentsUnavailable) firing: More than 20 segments have been unavailable for wmf_netflow on the druid_analytics Druid cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid/Alerts#Druid_Segments_Unavailable - https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&var-cluster=druid_analytics&panelId=49&fullscreen&orgId=1&var-cluster=druid_analytics - https://alerts.wikimedia.org [00:38:18] (DruidSegmentsUnavailable) resolved: More than 30 segments have been unavailable for wmf_netflow on the druid_analytics Druid cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid/Alerts#Druid_Segments_Unavailable - https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&var-cluster=druid_analytics&panelId=49&fullscreen&orgId=1&var-cluster=druid_analytics - https://alerts.wikimedia.org [00:38:18] (DruidSegmentsUnavailable) resolved: More than 20 segments have been unavailable for wmf_netflow on the druid_analytics Druid cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid/Alerts#Druid_Segments_Unavailable - https://grafana.wikimedia.org/dashboard/db/druid?refresh=1m&var-cluster=druid_analytics&panelId=49&fullscreen&orgId=1&var-cluster=druid_analytics - https://alerts.wikimedia.org [08:53:51] (03CR) 10Joal: "Last minor changes before merge." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/739129 (https://phabricator.wikimedia.org/T258834) (owner: 10AKhatun) [09:49:05] I have merged this, to deal with the druid alerts: https://gerrit.wikimedia.org/r/c/operations/alerts/+/740128 [09:52:04] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Snapshot and Reload cassandra2 pageview_per_article data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) Commancing reloading of the 11th snapshot. ` ### Reloading table data in keyspace loca... [10:00:53] (03CR) 10AKhatun: Save commons json dumps as a table and add fields for wikidata (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/739129 (https://phabricator.wikimedia.org/T258834) (owner: 10AKhatun) [10:11:41] 10Data-Engineering, 10Data-Engineering-Kanban: Error creating custom SQL metrics in Superset (event_sanitized.centralnoticebannerhistory) - https://phabricator.wikimedia.org/T292751 (10BTullis) Hello again @EYener - Apologies if I've misunderstood, but you are //already// an owner of the `centralnoticebannerhi... [10:17:25] I'm going to be restarting the hive services today, as part of T295673 [10:17:25] Beginning in a moment with a restart of the `hive-server2` and `hive-metastore` services on the standby server: an-coord1002 [10:18:57] !log btullis@an-coord1002:~$ sudo systemctl restart hive-server2 hive-metastore [10:19:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:23:34] DNS change for hive to use the standby, if anyone would like to review: https://gerrit.wikimedia.org/r/c/operations/dns/+/740537 [10:43:25] elukey: Thanks for the review. :-) [10:44:48] (03CR) 10Joal: Save commons json dumps as a table and add fields for wikidata (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/739129 (https://phabricator.wikimedia.org/T258834) (owner: 10AKhatun) [10:44:50] !log deploying DNS change to switch hive to the standby server. [10:44:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:46:56] np :) [11:33:53] About to roll-restart the hive services on an-coord1001... [11:36:39] !log btullis@an-coord1001:~$ sudo systemctl restart hive-server2 hive-metastore [11:36:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:18:37] !log failed back the hive services to an-coord1001 via CNAME change [12:18:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:23:35] (03PS7) 10AKhatun: Save commons json dumps as a table and add fields for wikidata [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/739129 (https://phabricator.wikimedia.org/T258834) [12:27:49] (03CR) 10AKhatun: Save commons json dumps as a table and add fields for wikidata (038 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/739129 (https://phabricator.wikimedia.org/T258834) (owner: 10AKhatun) [12:36:28] 10Data-Engineering, 10Data-Engineering-Kanban: Error creating custom SQL metrics in Superset (event_sanitized.centralnoticebannerhistory) - https://phabricator.wikimedia.org/T292751 (10EYener) 05Open→03Resolved Thank you @BTullis ! Apologies for missing the ownership of `event_sanitized.` - I tried to edit... [12:37:20] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics: [REQUEST] Manipulate dataset in Superset - https://phabricator.wikimedia.org/T292262 (10EYener) 05Open→03Resolved Marking resolved as @BTullis was able to help us get this working. Thank you! [13:16:04] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Send some existing Gobblin metrics to prometheus - https://phabricator.wikimedia.org/T294420 (10fgiunchedi) Overall numbers look good to me @JAllemandou ! A few observations/notes: * Please consider appending units to the metr... [13:23:43] (03CR) 10Joal: [C: 03+2] "Merging!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/739129 (https://phabricator.wikimedia.org/T258834) (owner: 10AKhatun) [13:31:46] (03Merged) 10jenkins-bot: Save commons json dumps as a table and add fields for wikidata [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/739129 (https://phabricator.wikimedia.org/T258834) (owner: 10AKhatun) [13:36:11] 10Quarry, 10User-dcaro, 10cloud-services-team (Kanban): [quarry] Fancy up the CI pipeline in Jenkins - https://phabricator.wikimedia.org/T289569 (10dcaro) a:05dcaro→03None [13:49:05] (03PS1) 10Joal: Update wikidata_entity table create and oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/740589 (https://phabricator.wikimedia.org/T258834) [13:51:46] (03PS1) 10Joal: Add structured_data.commons_entity table create [analytics/refinery] - 10https://gerrit.wikimedia.org/r/740590 (https://phabricator.wikimedia.org/T258834) [14:39:48] 10Data-Engineering, 10Data-Engineering-Kanban: Error creating custom SQL metrics in Superset (event_sanitized.centralnoticebannerhistory) - https://phabricator.wikimedia.org/T292751 (10BTullis) OK, thanks @EYener - Good to know. [14:55:20] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10User-razzi: Presto error in Superset - https://phabricator.wikimedia.org/T292879 (10BTullis) I think that we should either 1) make another ticket or 2) re-title this ticket to follow this up by fixing the LDAP integration and Superset user accou... [15:16:44] o/ [15:25:34] mforns: helLooooOOo :) [15:27:27] helooooo, supp? [15:27:31] ottomata: ^ [15:28:15] hello! [15:28:32] wanna check in real quick about code stuff? [15:29:52] ottomata: yes! bc? [15:30:36] k! [15:34:50] (03CR) 10AKhatun: "Just some keyword changes" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/740589 (https://phabricator.wikimedia.org/T258834) (owner: 10Joal) [15:37:44] (03CR) 10AKhatun: Add structured_data.commons_entity table create (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/740590 (https://phabricator.wikimedia.org/T258834) (owner: 10Joal) [15:57:51] 10Analytics, 10Analytics-Dashiki: Dashboard Directory research: Look at Hay's directory - https://phabricator.wikimedia.org/T99675 (10odimitrijevic) 05Open→03Declined [15:58:11] 10Analytics, 10Analytics-Dashiki: Update label in Vital Signs - https://phabricator.wikimedia.org/T86600 (10odimitrijevic) 05Open→03Declined [15:58:26] 10Analytics, 10Analytics-Dashiki: Analyst bookmarks Vital Signs showing multiple metrics - https://phabricator.wikimedia.org/T86966 (10odimitrijevic) 05Open→03Declined [16:12:14] (03PS2) 10Joal: Update wikidata_entity table create and oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/740589 (https://phabricator.wikimedia.org/T258834) [16:29:17] (03PS2) 10Joal: Add structured_data.commons_entity table create [analytics/refinery] - 10https://gerrit.wikimedia.org/r/740590 (https://phabricator.wikimedia.org/T258834) [16:31:08] (03CR) 10Joal: "I think that top-level field in hive are case insensitive, but this is good for consistency :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/740589 (https://phabricator.wikimedia.org/T258834) (owner: 10Joal) [16:31:25] (03CR) 10Joal: Add structured_data.commons_entity table create (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/740590 (https://phabricator.wikimedia.org/T258834) (owner: 10Joal) [16:49:33] elukey: I came by this today - seems interesting! https://microsoft.github.io/SynapseML/ [16:54:41] nice thanks! [17:09:37] 10Analytics-Radar, 10Data-Engineering, 10Event-Platform: Move Kafka Jumbo's TLS clients to the new bundle - https://phabricator.wikimedia.org/T296064 (10odimitrijevic) p:05Triage→03Medium [17:19:13] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Browser-Support-Microsoft-Edge: Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10odimitrijevic) a:03EChetty [17:24:09] 10Analytics-Radar, 10Product-Analytics: Develop a consistent rule for which special pages count as pageviews - https://phabricator.wikimedia.org/T240676 (10ldelench_wmf) a:03Iflorez [17:25:57] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Metrics-Platform, 10Browser-Support-Microsoft-Edge: Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10EChetty) a:05EChetty→03jlinehan [17:26:07] 10Analytics, 10Product-Analytics, 10Epic: Revamp analytics.wikimedia.org data portal & landing page - https://phabricator.wikimedia.org/T253393 (10ldelench_wmf) [17:27:30] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering, 10Product-Analytics: Support including edits to deleted pages in editing metrics - https://phabricator.wikimedia.org/T295212 (10odimitrijevic) p:05Triage→03Low [17:41:45] heya mforns - while investigating netflow in druid for data retention I found 2 days not having been re-indexed with sanitized data :S [17:42:15] in meeting joal , will look [17:42:35] mforns: I'm gonna create a task and point it to you [17:52:27] 10Data-Engineering: Reindex netflow data before 2020-09-04 to sanitize dimensions - https://phabricator.wikimedia.org/T296206 (10JAllemandou) [17:52:45] mforns: --^ the problem is actually broader than I had imagined - let's talk about that with the team [17:53:02] joal: ok! [17:54:04] joal: it is possible that this was me forgetting to re-run some dates!?! [17:54:15] I dont' think so mforns [17:54:38] 10Analytics-Radar, 10Product-Analytics: Set up a system for team-managed command-line jobs - https://phabricator.wikimedia.org/T271420 (10mpopov) 05Open→03Resolved a:03mpopov This work was done in {T291957} [17:57:09] 10Data-Engineering: Review druid deep-storage making sure that old segments having been reindexed are deleted - https://phabricator.wikimedia.org/T296207 (10JAllemandou) [18:16:53] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Desktop Improvements, and 3 others: Add agent_type and access_method to sticky header instrumentation - https://phabricator.wikimedia.org/T294246 (10LGoto) a:03nray [19:04:54] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering, 10Product-Analytics: Support including edits to deleted pages in editing metrics - https://phabricator.wikimedia.org/T295212 (10JAllemandou) Thank you for bringing this up @nshahquinn-wmf :) This topic has been discussed when we built wikistats2, and... [19:07:20] (03CR) 10Clare Ming: [C: 03+2] Elaborate on reading depth schema fields [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/740287 (https://phabricator.wikimedia.org/T294777) (owner: 10Nray) [19:08:44] (03Merged) 10jenkins-bot: Elaborate on reading depth schema fields [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/740287 (https://phabricator.wikimedia.org/T294777) (owner: 10Nray) [19:19:51] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Metrics-Platform, 10Browser-Support-Microsoft-Edge: Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10Ratte) Same problem in ru.wikisource. When trying to create a page in a [[ https://en.wikisou... [20:24:34] a-team: didn't we find a solution to refine monitor alerts for delayed sanitization of datasets with gaps? [20:53:55] 10Quarry, 10User-dcaro, 10cloud-services-team (Kanban): [quarry] Fancy up the CI pipeline in Jenkins - https://phabricator.wikimedia.org/T289569 (10Andrew) 05Open→03Resolved a:03Andrew I think this is done -- we use the newer pipeline and Jenkins even runs some tests. [21:16:36] 10Data-Engineering, 10Data-Engineering-Kanban, 10User-razzi: Increase Superset Timeout - https://phabricator.wikimedia.org/T294771 (10razzi) The setting we changed appears to apply only to SQL lab; the superset dashboards continue to time out after 1 minute {F34762283} [23:40:51] 10Analytics, 10Patch-For-Review, 10Product-Analytics (Kanban): Add mediawiki_skin_diff to the allowlist - https://phabricator.wikimedia.org/T287255 (10jwang) Hi @mforns, could you review the patch?