[00:00:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.09, 8.16, 8.54 [00:00:01] gn rhinos [00:00:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.28, 2.70 [00:01:11] @Bukkit: me sleeping is going well [00:01:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.99, 7.87, 8.50 [00:02:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.35, 7.76, 8.33 [00:02:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.08, 2.70 [00:02:27] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:02:38] huh [00:02:43] why icinga-miraheze [00:02:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.48, 3.24 [00:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:03:57] paladox: you able to monitor puppet or should I log back in [00:04:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.22, 9.00, 8.31 [00:04:13] I'm busy with something else [00:04:20] I will look [00:04:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.64, 8.27, 7.80 [00:06:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.57, 8.84, 8.35 [00:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:06:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.73, 3.38 [00:06:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.19, 9.32, 8.24 [00:07:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.38, 2.88 [00:07:34] it passed [00:07:48] * RhinosF1 off to sleep - take 2 [00:08:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.60, 8.04, 8.12 [00:08:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.80, 3.18 [00:08:28] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:08:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.40, 8.72, 8.14 [00:10:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 8.49, 8.49 [00:11:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.13, 8.85, 8.48 [00:11:10] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.17, 4.62, 3.58 [00:13:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.66, 8.95, 8.57 [00:13:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.17, 4.55, 3.51 [00:14:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.31, 8.34, 8.44 [00:16:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.69, 8.59, 8.45 [00:17:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.88, 3.44 [00:18:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 9.01, 8.70 [00:18:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 4.00, 3.69 [00:18:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 4.00, 3.99 [00:18:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.89, 8.00, 8.25 [00:19:02] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [00:20:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 9.46, 8.91 [00:20:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.37, 9.63, 8.82 [00:21:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.99, 3.29, 3.29 [00:22:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 9.27, 8.92 [00:22:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.45, 9.12, 8.73 [00:22:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.34, 10.25, 9.12 [00:22:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.19, 9.43, 8.98 [00:23:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.77, 8.86, 8.44 [00:24:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.18, 9.62, 8.97 [00:25:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.24, 8.07, 8.19 [00:26:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.60, 4.40, 4.01 [00:26:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.74, 9.76, 9.20 [00:26:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.08, 9.17, 8.89 [00:26:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.99, 9.20, 8.98 [00:28:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.43, 3.81, 3.85 [00:28:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.08, 9.78, 9.21 [00:29:22] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.89, 1.54, 1.18 [00:30:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.87, 2.89, 3.30 [00:30:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.04, 9.86, 9.22 [00:31:22] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.67, 1.60, 1.24 [00:32:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.68, 3.75 [00:32:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.82, 9.64, 9.21 [00:33:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.70, 8.92, 8.38 [00:34:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.93, 9.82, 9.31 [00:35:22] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.87, 1.80, 1.41 [00:36:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.24, 3.31 [00:36:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.46, 9.31, 9.31 [00:37:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.69, 8.16, 8.21 [00:38:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.38, 3.64, 3.45 [00:38:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.46, 9.62, 9.33 [00:39:22] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.14, 1.93, 1.56 [00:40:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.38, 3.36, 3.36 [00:41:22] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.96, 1.94, 1.61 [00:42:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.60, 8.05, 8.50 [00:43:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.76, 9.41, 8.70 [00:43:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.35, 9.46, 8.68 [00:43:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.54, 3.83 [00:48:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.61, 9.93, 9.37 [00:49:22] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.05, 1.98, 1.76 [00:49:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.82, 9.82, 9.15 [00:49:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.32, 9.80, 9.14 [00:52:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.92, 9.60, 9.41 [00:53:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.23, 9.69, 9.22 [00:53:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.52, 3.59 [00:54:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.52, 7.86, 8.44 [00:55:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.37, 9.30, 9.14 [00:55:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.39, 3.10, 3.42 [00:57:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.99, 3.08, 3.37 [00:58:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.26, 7.66, 8.44 [00:59:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.79, 9.70, 9.31 [00:59:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.14, 7.83, 8.39 [01:00:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.66, 8.47, 8.24 [01:02:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.12, 8.02, 8.11 [01:04:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 8.98, 8.62 [01:04:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.78, 1.84, 1.41 [01:05:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.24, 9.02, 9.28 [01:06:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.39, 8.43, 8.46 [01:06:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.71, 1.43, 1.31 [01:08:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.55, 9.03, 8.92 [01:09:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.25, 8.78, 8.52 [01:11:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.48, 9.58, 8.76 [01:12:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.86, 9.16, 9.02 [01:13:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.17, 4.21, 3.39 [01:13:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.09, 9.61, 8.89 [01:13:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.90, 8.45, 8.49 [01:14:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.31, 9.58, 9.19 [01:17:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 4.04, 3.54 [01:18:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.78, 8.91, 8.46 [01:18:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.43, 3.63, 3.45 [01:19:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.33, 3.41, 3.37 [01:21:22] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.39, 1.23, 1.96 [01:22:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.48, 8.94, 9.21 [01:22:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.04, 3.37, 3.39 [01:23:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 4.00, 3.60 [01:23:22] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.01, 1.53, 1.98 [01:23:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.35, 8.15, 8.50 [01:24:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.52, 3.47 [01:25:22] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.19, 1.38, 1.87 [01:25:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.01, 8.03, 8.49 [01:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:27:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.57, 8.32, 8.53 [01:28:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.96, 8.40, 8.41 [01:28:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.96, 3.17, 3.34 [01:29:22] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.93, 1.13, 1.65 [01:31:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.98, 8.27, 8.48 [01:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:36:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.79, 7.80, 8.43 [01:38:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.71, 3.40 [01:40:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.54, 3.38 [01:41:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.59, 9.30, 8.67 [01:42:16] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.89, 8.64, 8.31 [01:43:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.69, 8.99, 8.63 [01:44:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 3.11, 3.27 [01:45:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.77, 9.01, 8.68 [01:45:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.41, 8.82 [01:46:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.21, 9.31, 8.65 [01:46:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.59, 8.58, 8.33 [01:47:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.06, 9.40, 8.87 [01:47:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.58, 8.19, 7.87 [01:47:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.34, 9.15, 8.79 [01:48:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.81, 9.21, 8.70 [01:48:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.29, 8.48, 8.33 [01:48:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.72, 3.52 [01:49:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 7.78, 7.76 [01:49:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.33, 9.01, 8.57 [01:49:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.80, 8.04, 8.41 [01:50:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.65, 9.70, 8.94 [01:51:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.12, 9.34, 8.74 [01:52:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.57, 1.73, 1.28 [01:52:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 9.10, 8.57 [01:53:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.32, 8.58, 8.53 [01:54:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.45, 1.56, 1.27 [01:56:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.22, 9.93, 9.34 [01:57:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.88, 9.44, 9.35 [01:59:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.40, 8.71, 8.54 [02:00:08] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.59, 8.12, 8.38 [02:00:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.55, 3.51 [02:00:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.84, 7.78, 8.26 [02:02:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:02:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.07, 9.58, 9.42 [02:03:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.78, 7.60, 8.13 [02:04:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.86, 9.37, 8.80 [02:04:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.73, 8.83, 9.16 [02:04:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 3.19, 3.40 [02:06:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 9.59, 8.96 [02:07:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:08:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.13, 8.43, 8.83 [02:08:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.98, 3.99, 3.67 [02:09:46] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.57, 7.55, 8.33 [02:10:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.37, 7.62, 8.27 [02:10:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.18, 8.01, 8.64 [02:10:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.64, 3.59 [02:12:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:12:37] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.67, 7.56, 8.38 [02:12:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.67, 3.13, 3.39 [02:13:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.67, 8.82, 8.70 [02:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:16:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.52, 3.51 [02:20:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.69, 8.65, 8.51 [02:21:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.39, 9.33, 8.94 [02:22:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.89, 8.47, 8.47 [02:22:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.18, 2.87, 3.27 [02:23:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.50, 9.04, 8.86 [02:23:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.93, 9.01, 8.25 [02:23:52] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [02:23:52] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:26:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.79, 8.53, 8.53 [02:27:25] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.46, 7.30, 8.20 [02:27:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.62, 8.08, 8.04 [02:28:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.85, 7.50, 8.15 [02:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:31:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.72, 9.20, 8.45 [02:34:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:35:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.02, 9.19, 8.60 [02:39:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.81, 8.55, 7.85 [02:42:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.59, 9.22, 8.57 [02:43:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.29, 8.65, 8.07 [02:44:05] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.12, 9.36, 8.68 [02:45:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.48, 9.34, 8.89 [02:46:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.16, 8.94, 8.61 [02:46:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.93, 8.13, 8.01 [02:47:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.27, 9.16, 8.40 [02:51:03] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.99, 5.71, 4.50 [02:51:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.32, 9.18, 8.66 [02:53:03] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.09, 5.09, 4.42 [02:53:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.78, 8.48, 8.44 [02:53:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.68, 9.92, 9.50 [02:53:51] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [02:53:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.01, 9.50, 8.98 [02:54:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.57, 4.01, 3.31 [02:55:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.81, 9.33, 8.96 [02:56:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.32, 2.04, 1.68 [02:56:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.36, 3.16 [02:57:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.60, 9.18, 8.77 [02:58:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.80, 1.59, 1.55 [03:01:03] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.15, 7.93, 8.39 [03:01:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.76, 9.56, 9.40 [03:01:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.06, 8.81, 8.74 [03:03:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.98, 8.45, 9.01 [03:03:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.83, 8.78, 8.74 [03:06:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.65, 8.95, 8.12 [03:07:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.97, 7.92, 8.44 [03:08:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.97, 8.45, 8.03 [03:10:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.79, 8.62, 8.42 [03:11:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.49, 7.60, 8.39 [03:11:33] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.29, 9.18, 8.81 [03:12:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.65, 9.53, 8.70 [03:14:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.37, 8.31, 8.34 [03:15:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.87, 9.38, 9.03 [03:16:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.66, 9.45, 8.58 [03:17:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.75, 9.79, 9.21 [03:18:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.03, 8.67, 8.41 [03:19:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.58, 9.57, 9.19 [03:19:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.18, 8.64, 8.50 [03:19:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.77, 9.45, 9.07 [03:20:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.76, 7.71, 8.09 [03:20:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.96, 4.68, 2.60 [03:21:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.84, 10.29, 9.41 [03:22:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 8.97, 8.58 [03:22:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 8.89, 8.24 [03:23:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.70, 9.88, 9.37 [03:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:25:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.35, 9.03, 9.13 [03:25:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.76, 8.11, 8.39 [03:26:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.72, 9.22, 8.58 [03:26:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.39, 8.48, 8.45 [03:26:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.31, 1.86, 1.99 [03:27:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.09, 9.90, 9.59 [03:28:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.23, 8.94, 8.57 [03:30:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.30, 1.78, 1.90 [03:30:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.91, 8.08, 8.23 [03:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:32:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.33, 8.50, 8.46 [03:32:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.60, 1.82 [03:34:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.48, 1.85, 1.88 [03:34:51] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [03:36:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.87, 1.82, 1.87 [03:36:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:37:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.88, 7.53, 8.44 [03:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:40:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.16, 1.29, 1.63 [03:41:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.04, 9.14, 8.90 [03:41:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 9.36, 9.11 [03:41:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.58, 3.33 [03:43:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.10, 8.88, 8.97 [03:43:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.56, 2.88, 3.10 [03:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:45:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.89, 9.58, 9.17 [03:46:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.12, 9.42, 8.85 [03:47:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.22, 8.76, 8.43 [03:48:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.61, 8.23, 8.47 [03:48:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.22, 8.36, 8.31 [03:53:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.27, 7.69, 8.39 [03:57:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.32, 7.33, 8.24 [04:04:55] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [04:04:55] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406957149505615 seconds [04:11:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.61, 8.22, 7.66 [04:12:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:13:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.53, 9.23, 8.42 [04:13:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.00, 7.82, 7.59 [04:15:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 8.48, 8.25 [04:16:34] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.76, 1.84, 1.18 [04:16:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.97, 8.32, 7.84 [04:18:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.18, 8.18, 7.84 [04:18:33] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.94, 1.44, 1.12 [04:22:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.41, 9.82, 8.56 [04:22:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:23:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.28, 9.08, 8.51 [04:24:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.52, 8.91, 8.41 [04:24:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 8.81, 7.97 [04:25:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 8.76, 8.44 [04:26:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.77, 7.91, 7.73 [04:26:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.92, 7.93, 8.17 [04:29:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.05, 9.48, 8.73 [04:31:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.79, 9.70, 8.91 [04:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:37:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.00, 7.89, 8.35 [04:41:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.49, 8.65, 8.08 [04:43:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:43:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.22, 9.04, 8.28 [04:45:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.79, 8.75, 8.26 [04:47:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.28, 9.23, 8.49 [04:49:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.29, 9.44, 8.67 [04:50:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:51:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.90, 8.92, 7.92 [04:51:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.06, 10.09, 9.01 [04:52:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.50, 8.85, 8.02 [04:53:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.73, 8.56, 7.92 [04:53:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.67, 9.79, 9.02 [04:55:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.99, 8.04, 7.80 [04:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:55:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 8.63, 8.38 [04:56:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.71, 8.84, 8.31 [04:56:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.03, 8.23, 7.95 [04:57:44] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.48, 8.04, 8.21 [05:02:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.64, 8.50, 8.37 [05:03:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.76, 8.85, 8.52 [05:03:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.26, 7.97, 8.42 [05:05:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:07:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.06, 8.96, 8.76 [05:09:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.18, 8.26, 8.39 [05:12:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.37, 8.54, 8.26 [05:14:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.35, 2.84 [05:16:03] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.16, 7.69, 8.00 [05:16:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 3.75, 3.04 [05:18:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.52, 3.04 [05:20:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 2.06, 1.45 [05:20:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.67, 3.28, 3.01 [05:24:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.77, 1.49 [05:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:26:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.43, 1.64, 1.47 [05:26:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [05:26:43] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:29:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.30, 8.14, 8.49 [05:31:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.56, 8.26, 7.80 [05:32:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.12, 8.84, 8.22 [05:33:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.91, 7.54, 7.59 [05:34:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.90, 8.72, 8.25 [05:34:53] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [05:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:38:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 7.90, 8.03 [05:38:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.21, 8.64, 8.30 [05:40:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.90, 8.36, 8.24 [05:41:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.69, 8.56, 7.78 [05:42:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.42, 8.54, 8.33 [05:43:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.69, 8.24, 7.76 [05:44:25] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.36, 8.96, 8.48 [05:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:46:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.27, 7.91, 8.16 [05:55:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.74, 1.34 [05:56:42] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [05:57:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.45, 1.27 [06:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:04:55] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [06:04:55] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406552791595459 seconds [06:06:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.52, 8.50, 8.00 [06:08:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.89, 8.41, 8.03 [06:10:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:12:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.13, 8.68, 8.16 [06:13:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:13:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.59, 7.60, 7.19 [06:14:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.14, 8.03, 7.99 [06:15:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.33, 7.67, 7.26 [06:18:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:20:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.26, 8.58, 8.18 [06:21:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.28, 9.66, 8.25 [06:22:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:22:47] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:23:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.96, 9.56, 8.40 [06:25:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 8.36, 8.07 [06:26:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.74, 7.97, 8.05 [06:27:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.15, 9.58, 8.64 [06:27:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.35, 7.94, 7.95 [06:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:29:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.22, 8.49, 8.35 [06:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:36:35] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=luigismansionwiki --username-prefix=luigismansion --report 1 /home/reception/Luigi's+Mansion+Wiki-20220216161709.xml (START) [06:36:36] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=luigismansionwiki --username-prefix=luigismansion --report 1 /home/reception/Luigi's+Mansion+Wiki-20220216161709.xml (END - exit=512) [06:36:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:36:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.83, 9.21, 8.17 [06:37:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:39:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.92, 8.56, 8.06 [06:41:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.87, 7.85, 7.85 [06:42:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.12, 9.03, 8.46 [06:43:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.89, 8.53, 8.28 [06:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:45:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.22, 8.07, 8.15 [06:46:34] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.48, 8.13, 8.21 [06:50:45] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:54:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 8.56, 8.29 [06:56:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.31, 7.71, 8.01 [07:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:04:54] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [07:16:22]