[04:38:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [05:04:16] (EventgateValidationErrors) firing: ... [05:04:16] eventgate-analytics-external stream eventlogging_SearchSatisfaction validation errors detected in past 15 min - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos - https://alerts.wikimedia.org/?q=alertname%3DEventgateValidationErrors [05:19:16] (EventgateValidationErrors) resolved: ... [05:19:16] eventgate-analytics-external stream eventlogging_SearchSatisfaction validation errors detected in past 15 min - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos - https://alerts.wikimedia.org/?q=alertname%3DEventgateValidationErrors [07:13:30] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [12:53:25] 10Data-Engineering: [NEEDS GROOMING] deequ repo should be instantiated from Wikimedia's DQ metrics store - https://phabricator.wikimedia.org/T353939 (10gmodena) [12:56:36] 10Data-Engineering: [NEEDS GROOMING] we should provide DQ integration with Python - https://phabricator.wikimedia.org/T353940 (10gmodena) [13:06:10] 10Data-Engineering: [NEEDS GROOMING] Define, document and enforce best pratices for instrumenting DQ pipelines - https://phabricator.wikimedia.org/T353941 (10gmodena) [13:22:10] 10Analytics-Radar, 10Data-Engineering, 10Pageviews-API, 10Tool-Pageviews: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10TheDJ) @MusikAnimal is this still an issue ? Since there hasn't happened anything in this ticket for 3 years (if you ignore th... [14:53:30] 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Service implementation for wdqs10[17-21] - https://phabricator.wikimedia.org/T351671 (10bking) 05Open→03Resolved I can confirm that all hosts are active in pybal and their data is loaded. Closing... [15:27:51] 10Data-Engineering, 10Data-Platform-SRE, 10Product-Analytics: Conda analytics environments breakage - conflicting dependencies between r-base and other - https://phabricator.wikimedia.org/T343823 (10mpopov) @nettrom_WMF: When you're back from holidays, can you please try Ben's pinning solution and see if tha... [18:36:35] 10Data-Engineering, 10Data Products (Data Products Sprint 05): Make defaults immutable for Airflow confs - https://phabricator.wikimedia.org/T325014 (10xcollazo) [19:25:48] 10Data-Engineering: Airflow DAG mediawiki_history_denormalize failed with NPE - https://phabricator.wikimedia.org/T350489 (10xcollazo) 05Open→03Resolved [19:33:47] (03PS1) 10Mforns: Make traffic anomaly detection query robust vs. MaxMind updates [analytics/refinery] - 10https://gerrit.wikimedia.org/r/985333 (https://phabricator.wikimedia.org/T353956) [19:34:56] (03CR) 10Mforns: [V: 03+2] "I tested this successfully :]" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/985333 (https://phabricator.wikimedia.org/T353956) (owner: 10Mforns) [20:25:16] (03CR) 10Mforns: [V: 03+2 C: 03+2] Make traffic anomaly detection query robust vs. MaxMind updates [analytics/refinery] - 10https://gerrit.wikimedia.org/r/985333 (https://phabricator.wikimedia.org/T353956) (owner: 10Mforns) [21:26:48] !log re-ran Airflow job anomaly_detection_traffic_distribution_daily from 2023-12-14 to 2023-12-21 [21:26:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:29:47] !log re-ran the Airflow DAG unique_devices_per_domain_daily for 2023-12-14 [21:29:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:30:14] !log re-ran the Airflow DAG unique_devices_per_project_family_daily for 2023-12-14 [21:30:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:37:24] !log re-ran the Airflow DAG druid_load_unique_devices_per_project_family_daily for 2023-12-14 [21:37:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:37:50] !log re-ran the Airflow DAG druid_load_unique_devices_per_domain_daily for 2023-12-14 [21:37:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:38:22] !log re-ran the Airflow DAG cassandra_load_unique_devices_daily for 2023-12-14 [21:38:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log