[00:31:11] (03PS3) 10Klein Muçi: Fix typo [analytics/pivot/deploy] - 10https://gerrit.wikimedia.org/r/787785 (https://phabricator.wikimedia.org/T201491) [02:13:22] (03PS3) 10Jenniferwang: Bug: T299007 Add the mediawiki_reading_depth event platform stream to the allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/753178 (https://phabricator.wikimedia.org/T299007) [03:37:38] (03PS1) 10Jenniferwang: Bug: T299007 Add the mediawiki_reading_depth event platform stream to the allowlist. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/787840 (https://phabricator.wikimedia.org/T299007) [03:41:11] (03CR) 10Jenniferwang: "Hi Mforns," [analytics/refinery] - 10https://gerrit.wikimedia.org/r/787840 (https://phabricator.wikimedia.org/T299007) (owner: 10Jenniferwang) [03:51:07] (03CR) 10Jenniferwang: "Hi All," [analytics/refinery] - 10https://gerrit.wikimedia.org/r/753178 (https://phabricator.wikimedia.org/T299007) (owner: 10Jenniferwang) [04:00:42] 10Quarry: Kill all queries stuck in running or queued state - https://phabricator.wikimedia.org/T307263 (10GeoffreyT2000) [04:01:28] 10Quarry: Pressing the Stop button in Quarry results in a 500 error - https://phabricator.wikimedia.org/T290146 (10GeoffreyT2000) [04:01:30] 10Quarry: Kill all queries stuck in running or queued state - https://phabricator.wikimedia.org/T307263 (10GeoffreyT2000) [04:40:30] 10Quarry: Pressing the Stop button in Quarry results in a 500 error - https://phabricator.wikimedia.org/T290146 (10GeoffreyT2000) p:05High→03Medium Still not fixed after several months. [06:35:14] PROBLEM - Check unit status of mediawiki-history-drop-snapshot on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki-history-drop-snapshot https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:54:26] 10Quarry: Pressing the Stop button in Quarry results in a 500 error - https://phabricator.wikimedia.org/T290146 (10Certes) Yes, my trivial query 61115 claims to have be running for nearly four months now, and the Stop button gives error 500. Can we at least do a one-off task to stop all queries which have been... [11:12:04] 10Data-Engineering, 10Privacy Engineering, 10SRE-swift-storage: Swift for differential privacy data publication - https://phabricator.wikimedia.org/T307245 (10Peachey88) [14:26:48] PROBLEM - Check unit status of eventlogging_to_druid_navigationtiming_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_navigationtiming_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:29:06] 10Data-Engineering, 10Data-Persistence, 10Privacy Engineering, 10SRE-swift-storage: Swift for differential privacy data publication - https://phabricator.wikimedia.org/T307245 (10RhinosF1) [15:00:30] RECOVERY - Check unit status of eventlogging_to_druid_navigationtiming_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_navigationtiming_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [18:57:32] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [19:52:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [22:10:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [22:25:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [22:41:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [22:46:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage