[02:55:07] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [03:06:03] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [03:19:43] 10Quarry, 10cloud-services-team (Kanban): Quarry is degraded/partially inaccessible - https://phabricator.wikimedia.org/T290291 (10Chlod) 05Open→03Resolved Works like a charm. Thanks, @Andrew! [04:25:13] RECOVERY - Check unit status of monitor_refine_event_sanitized_analytics_delayed on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_event_sanitized_analytics_delayed https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [11:02:02] (03PS1) 10GoranSMilovanovic: rollout [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/718347 [11:02:04] (03PS1) 10GoranSMilovanovic: del WDMC Titles OLD [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/718348 [11:02:14] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] rollout [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/718347 (owner: 10GoranSMilovanovic) [11:02:27] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] del WDMC Titles OLD [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/718348 (owner: 10GoranSMilovanovic) [19:48:23] PROBLEM - Hadoop NodeManager on an-worker1140 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [19:59:25] RECOVERY - Hadoop NodeManager on an-worker1140 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process