[03:47:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [03:52:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [04:41:55] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) 05Open→03Resolved a:03Marostegui I have cleaned up `research_role2` and double checked that everything keeps working a... [04:50:37] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Persistence (Consultation), 10Data-Services, and 2 others: Toolforge db: View 'fiwiki_p.flaggedrevs' references invalid table/column/rights to use them - https://phabricator.wikimedia.org/T302233 (10Marostegui) what is pending here? [08:30:24] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for guwwiki - https://phabricator.wikimedia.org/T303761 (10BTullis) The views have been created on the clouddb hosts for this database. [08:31:22] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwikivoyage - https://phabricator.wikimedia.org/T302798 (10BTullis) Thanks for the explanation @bd808 So I'm happy that the views have been created on all of the clouddb databases. What else r... [08:34:35] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for guwwiki - https://phabricator.wikimedia.org/T303761 (10Marostegui) 05Open→03Resolved a:03Marostegui I can see the rows on `guwwiki_p` on all the s5 clouddb* hosts so resolving this Thanks... [08:35:01] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwikivoyage - https://phabricator.wikimedia.org/T302798 (10Marostegui) 05Open→03Resolved a:03Marostegui Tested shnwikivoyage_p and looks good. Resolving! [10:02:20] 10Data-Engineering: Check home/HDFS leftovers of keepit-ssh - https://phabricator.wikimedia.org/T306415 (10Snwachukwu) Hi @ahemmer, Kindly confirm that we can go ahead to delete the following 0 size hidden file in HDFS belonging to CherRaye Glenn: ` sandraebele@Sandras-MacBook-Pro Opsweek % sh check_user_files.... [10:39:42] I have a superset dashboard, how can I get other folks to be able to see it? [10:59:31] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset: Help with data that's not appearing on charts - https://phabricator.wikimedia.org/T301895 (10BTullis) I have raised an upstream bug report on the Superset GitHub project. [[https://github.com/apache/superset/issues/19880|{line... [11:00:52] addshore: what is the data source for your dashboard? [11:01:07] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset: Help with data that's not appearing on charts - https://phabricator.wikimedia.org/T301895 (10BTullis) a:03BTullis Removing from Ops Week as I will follow up on this task. [11:01:39] presto_analytics_hive https://superset.wikimedia.org/superset/dashboard/mwcli/ [11:02:05] in that case, the underlying data in hadoop need to be readable (at least by `analytics-privatedata-users` group), everyone who needs access needs to be a member of that group (SSH-less access is enough) and they need to be in wmf or nda LDAP group [11:02:24] https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Dashboards_in_Superset_/_Hive_interfaces_(like_Hue)_that_do_access_private_data is the level of access they need [11:04:42] the mwcli dashboard works for me, which means the data are readable. so, the group membership is the only thing needed IMO addshore [11:04:52] aaah right! :) thanks! [11:05:13] np [11:10:40] addshore: Nice looking dashboard 😎 [11:11:08] :D thanks! [11:11:55] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset: Help with data that's not appearing on charts - https://phabricator.wikimedia.org/T301895 (10BTullis) I have also tested the same chart on Superset version 1.4.2 and it behaves in the same way. [12:37:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [12:57:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [12:59:31] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset: Help with data that's not appearing on charts - https://phabricator.wikimedia.org/T301895 (10BTullis) [13:04:52] 10Data-Engineering, 10Airflow: Set up backups and monitoring of airflow instances - https://phabricator.wikimedia.org/T307102 (10Ottomata) [13:33:06] (03CR) 10Luke Bowmaker: [C: 03+2] Image Suggestions feature schema (0310 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/779052 (owner: 10Luke Bowmaker) [13:33:44] (03Merged) 10jenkins-bot: Image Suggestions feature schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/779052 (owner: 10Luke Bowmaker) [14:21:02] 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Implement pageviews endpoints - https://phabricator.wikimedia.org/T288296 (10FGoodwin) [14:21:04] 10Data-Engineering-Radar, 10API Platform, 10Platform Engineering Roadmap: Retroactively fix uses of fmt.Sprintf where no formatting occurs - https://phabricator.wikimedia.org/T305510 (10FGoodwin) 05In progress→03Resolved [14:55:17] (03PS1) 10Snwachukwu: Add new wikis to sqoop list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/787513 (https://phabricator.wikimedia.org/T304632) [15:09:54] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Improvements over artifacts cache - https://phabricator.wikimedia.org/T307115 (10Antoine_Quhen) [15:18:34] (03PS1) 10GoranSMilovanovic: house_cleaning [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/787517 [15:19:17] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] house_cleaning [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/787517 (owner: 10GoranSMilovanovic) [15:31:23] (03PS1) 10GoranSMilovanovic: license [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/787519 [15:31:36] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] license [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/787519 (owner: 10GoranSMilovanovic) [15:33:08] (03PS1) 10GoranSMilovanovic: engines_pckgs [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/787520 [15:33:59] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] engines_pckgs [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/787520 (owner: 10GoranSMilovanovic) [15:45:06] (03PS1) 10GoranSMilovanovic: fix_license [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/787524 [15:45:15] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] fix_license [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/787524 (owner: 10GoranSMilovanovic) [16:04:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [16:12:04] 10Data-Engineering-Kanban, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Reimage WMCS db proxies to Bullseye - https://phabricator.wikimedia.org/T298940 (10razzi) a:03razzi [16:36:35] 10Data-Engineering: Check home/HDFS leftovers of jdl - https://phabricator.wikimedia.org/T306412 (10Snwachukwu) Hello @SCherukuwada, Kindly confirm that we can go ahead to delete the following data belonging to Jason Linehan: ` sandraebele@Sandras-MacBook-Pro Opsweek % sh check_user_files.sh jdl -e ====== stat... [16:58:28] (03CR) 10Ottomata: [C: 03+1] "I'm mostly unfamiliar with this file but LGTM! +1 merge away!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/787513 (https://phabricator.wikimedia.org/T304632) (owner: 10Snwachukwu) [16:59:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [17:17:53] (03CR) 10Mforns: "Code looks great!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/785153 (https://phabricator.wikimedia.org/T300028) (owner: 10Snwachukwu) [22:21:01] 10Data-Engineering, 10Event-Platform, 10SRE, 10Traffic, and 2 others: Incident: Banner sampling leading to a relatively wide site outage (mostly esams) - https://phabricator.wikimedia.org/T303036 (10lmata) [22:29:02] 10Data-Engineering, 10Event-Platform, 10SRE, 10Traffic, and 2 others: Incident: 2022-03-4 Banner sampling leading to a relatively wide site outage (mostly esams) - https://phabricator.wikimedia.org/T303036 (10lmata)