[00:00:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.49, 9.97, 9.71 [00:01:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.45, 3.49 [00:01:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 9.20, 8.89 [00:03:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.85, 3.67 [00:04:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.84, 9.44, 9.49 [00:04:23] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 670 failures. Last run 3 minutes ago with 670 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_conntrack],File[/usr/lib/nagios/plugins/check_ferm],File[/etc/nagios/nrpe.cfg],File[/usr/lib/nagios/plugins/check_smart] [00:05:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.68, 3.64 [00:05:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.73, 8.96, 8.90 [00:06:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.10, 10.09, 9.72 [00:06:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.81, 3.46 [00:07:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 4.09, 3.80 [00:08:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.98, 9.78, 9.66 [00:08:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.64, 3.49, 3.39 [00:09:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.63, 9.35, 9.17 [00:10:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.32, 10.06, 9.78 [00:10:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.16, 3.40, 3.37 [00:12:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.79, 9.70, 9.72 [00:13:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 9.23, 9.16 [00:14:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.03, 10.37, 9.97 [00:15:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.15, 9.95, 9.43 [00:16:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.45, 9.96, 9.89 [00:17:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.64, 9.96, 9.28 [00:17:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.87, 9.37, 9.28 [00:18:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.48, 3.42 [00:20:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.09, 3.84, 3.54 [00:22:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.44, 10.07, 9.90 [00:22:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.78, 3.56 [00:23:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.42, 9.40, 9.36 [00:24:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.37, 4.72, 3.93 [00:25:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.89, 9.62, 9.42 [00:26:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.56, 9.91, 9.94 [00:27:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.21, 9.17, 9.03 [00:28:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.04, 10.25, 10.07 [00:29:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 9.49, 9.45 [00:29:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.45, 8.81, 8.92 [00:30:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.31, 9.56, 9.82 [00:32:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.63, 9.87, 9.89 [00:32:39] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.63, 9.72, 9.51 [00:33:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.79, 9.84, 9.32 [00:34:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.79, 9.17, 8.91 [00:36:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 9.46, 9.07 [00:37:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.75, 9.63, 9.33 [00:39:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.20, 9.68, 9.37 [00:41:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.90, 8.59, 7.88 [00:42:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 9.60, 9.20 [00:43:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.34, 9.27, 9.57 [00:44:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 9.34, 9.15 [00:45:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 9.79, 9.64 [00:47:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.14, 7.58, 7.70 [00:49:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.02, 9.95, 9.71 [00:50:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.79, 9.37 [00:50:26] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.79, 1.51, 1.20 [00:51:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.47, 9.72, 9.66 [00:52:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.68, 1.23, 1.13 [00:52:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.79, 9.84, 9.45 [00:54:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.03, 9.20, 9.82 [00:56:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.37, 9.72, 9.94 [00:57:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 9.43, 9.47 [00:59:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.80, 9.82, 9.63 [01:00:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 9.82, 9.99 [01:04:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.27, 10.20, 9.52 [01:06:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 10.05, 9.91 [01:06:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.01, 9.60, 9.37 [01:08:28] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [01:09:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.92, 9.00, 9.03 [01:09:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.75, 8.87, 8.79 [01:10:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.28, 3.89 [01:13:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.24, 9.82, 9.27 [01:15:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.76, 10.03, 9.41 [01:16:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.79, 9.54, 9.90 [01:17:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.46, 9.72, 9.39 [01:18:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.51, 10.52, 10.23 [01:18:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 3.54, 3.76 [01:19:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.94, 1.76, 1.34 [01:21:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.63, 1.29, 1.22 [01:22:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.73, 9.81, 9.41 [01:22:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.73, 3.75, 3.80 [01:23:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.39, 9.51, 9.36 [01:25:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.21, 9.54, 9.65 [01:25:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.72, 9.30, 9.31 [01:27:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.76, 9.71 [01:27:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.02, 9.64, 9.44 [01:28:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 9.73, 9.54 [01:29:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.06, 9.51, 9.42 [01:31:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.85, 9.79, 9.80 [01:32:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.90, 9.62, 9.99 [01:32:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.87, 3.78 [01:33:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.00, 10.12, 9.66 [01:34:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.78, 9.67, 9.94 [01:34:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.88, 3.68, 3.70 [01:34:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.16, 9.75, 9.52 [01:35:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 9.78, 9.60 [01:36:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.66, 3.67 [01:37:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.34, 9.56 [01:37:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.27, 10.09, 9.74 [01:38:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 9.62, 9.52 [01:39:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.24, 8.92, 9.39 [01:39:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.37, 9.23, 9.48 [01:40:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.43, 3.58 [01:43:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.19, 9.49, 9.52 [01:44:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.78, 9.05, 9.76 [01:44:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.72, 3.64 [01:45:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.11, 9.04, 9.33 [01:46:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.16, 9.87, 9.97 [01:47:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.58, 8.83, 9.04 [01:48:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.77, 3.69 [01:49:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.52, 3.95 [01:50:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.58, 9.60, 9.87 [01:51:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.74, 9.87, 9.51 [01:52:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.06, 9.57, 9.81 [01:53:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.43, 3.86 [01:53:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.85, 10.19, 9.66 [01:53:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.20, 8.67, 8.86 [01:55:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.78, 9.57, 9.50 [01:55:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.94, 9.00, 8.95 [01:57:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 3.62, 3.79 [01:59:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.58, 3.75 [02:00:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.75, 3.65 [02:01:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.33, 4.22, 3.97 [02:02:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.31, 9.44, 9.82 [02:02:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.53, 3.59 [02:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:04:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.83, 9.79, 9.89 [02:05:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.82, 3.89 [02:06:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.55, 9.86, 9.92 [02:07:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.88, 4.04, 3.97 [02:11:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.81, 3.91 [02:11:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.27, 9.05, 9.05 [02:11:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.19, 9.06, 8.82 [02:13:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.19, 4.26, 4.05 [02:13:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.05, 8.71, 8.93 [02:13:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.06, 8.21, 8.54 [02:14:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.63, 8.02, 8.43 [02:15:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.69, 7.83, 8.34 [02:18:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.13, 7.24, 8.44 [02:18:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.47, 2.91, 3.36 [02:19:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.43, 3.82 [02:19:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 8.69, 8.74 [02:21:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.50, 3.51 [02:21:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.29, 8.24, 8.57 [02:23:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.10, 7.74, 8.34 [02:24:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.34, 3.35 [02:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:30:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.91, 3.63 [02:31:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.47, 2.53, 3.26 [02:32:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.41, 2.97, 3.31 [02:35:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 8.27, 7.88 [02:35:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.96, 3.93 [02:37:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.59, 7.51, 7.64 [02:37:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 4.20, 4.02 [02:45:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.16, 3.32, 3.83 [02:49:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.84, 3.52, 3.74 [02:49:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:57:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.25, 3.06 [02:58:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.49, 8.25, 7.59 [03:00:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.87, 7.66, 7.45 [03:01:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.12, 3.06 [03:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:05:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 2.87, 1.92 [03:06:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.00, 8.84, 8.02 [03:09:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.65, 1.94, 1.72 [03:10:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.93, 8.17, 7.95 [03:11:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.41, 2.04, 1.76 [03:11:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.88, 3.25, 3.85 [03:13:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.51, 1.60 [03:14:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.81, 8.58, 7.68 [03:16:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.15, 8.39, 7.72 [03:19:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.39, 9.19, 8.16 [03:21:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.39, 3.07 [03:22:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.79, 8.80, 8.36 [03:23:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.28, 3.60, 3.58 [03:23:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.45, 9.72, 8.67 [03:23:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.67, 3.21 [03:24:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.01, 8.38, 8.27 [03:25:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.36, 3.50 [03:25:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.52, 8.50, 8.35 [03:25:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.13, 3.38, 3.15 [03:29:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.26, 4.57, 3.89 [03:31:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 4.08, 3.47 [03:33:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.55, 3.33 [03:33:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:35:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 3.15, 3.21 [03:43:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.77, 3.59, 3.98 [03:45:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.91, 3.91, 4.06 [03:46:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:47:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.44, 3.18 [03:49:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 3.08, 3.08 [03:49:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.73, 3.25, 3.81 [03:51:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:55:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.50, 3.96, 3.40 [03:55:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.41, 3.67 [03:57:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.11, 3.39, 3.27 [03:57:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.45, 3.65 [03:59:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.47, 3.62 [03:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:01:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.37, 3.35, 3.56 [04:08:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:09:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.55, 4.06, 3.74 [04:13:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.91, 3.55, 3.32 [04:13:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.91, 3.73, 3.07 [04:13:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:15:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 3.22, 3.22 [04:15:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.52, 3.26, 2.98 [04:15:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.94, 3.90 [04:19:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 4.00, 3.91 [04:21:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.05, 3.22, 3.64 [04:25:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 2.71, 3.32 [04:59:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:00:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.14, 2.86 [05:02:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.79, 3.15 [05:04:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.62, 3.16 [05:04:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:08:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.78, 3.32 [05:12:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.86, 3.50 [05:16:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.72, 3.50 [05:18:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.55, 3.46 [05:22:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.14, 3.18, 3.31 [05:34:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:39:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:00:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.46, 2.96 [06:02:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.66, 3.12, 2.89 [06:12:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [06:41:37] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [06:45:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 4.14, 3.12 [06:47:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.95, 3.47, 3.00 [06:51:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.64, 2.65, 2.77 [06:54:37] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.06, 1.88, 1.57 [06:56:32] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.71, 1.79, 1.57 [07:00:20] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.38, 1.64, 1.55 [07:13:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [07:13:10] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:42:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:00:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.66, 3.21 [08:02:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.51, 3.18 [08:06:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.15, 3.37, 3.21 [08:12:16] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [08:30:55] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.32, 6.95, 5.13 [08:34:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.50, 6.50, 5.44 [09:08:43] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/ [09:08:43] dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query mineland.eu. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:59:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.00, 2.49 [10:01:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.45, 2.89, 2.52 [10:08:26] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp20.miraheze.org - CNAME OK [10:16:29] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [10:16:29] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:46:02] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [10:49:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.16, 2.46 [10:53:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.27, 2.69 [11:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:06:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 3.81, 3.01 [11:08:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.80, 3.15, 2.87 [11:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:17:36] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:22:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:26:16] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thunis.eu All nameservers failed to answer the query. [11:27:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:29:10] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/lib [11:29:10] /dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query fc.songcontests.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:39:58] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mineland.eu All nameservers failed to answer the query. [11:40:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:45:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:47:09] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [11:47:09] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:07:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:24] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3b18af6aee82...0b477c9c532c [12:09:25] [02miraheze/MirahezeMagic] 07translatewiki 030b477c9 - Localisation updates from https://translatewiki.net. [12:09:49] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/ [12:09:50] dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query mineland.eu. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:09:54] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.80, 1.39, 0.87 [12:10:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 8.77, 7.65 [12:11:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.13, 8.78, 7.73 [12:11:54] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.52, 1.44, 0.96 [12:13:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.66, 8.02, 7.59 [12:14:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.67, 8.18, 7.71 [12:16:42] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [12:17:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:19:22] miraheze/MirahezeMagic - translatewiki the build passed. [12:23:54] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.12, 1.86, 1.42 [12:24:17] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: SSL OK - wiki.thunis.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:25:54] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.28, 1.66, 1.40 [12:36:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.29, 8.63, 7.95 [12:38:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.01, 7.48, 7.61 [12:39:42] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:43:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:48:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:51:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.23, 8.79, 7.92 [12:51:54] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.03, 1.67, 1.49 [12:52:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.30, 2.94 [12:54:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.10, 2.91 [12:55:54] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.63, 1.80, 1.60 [12:56:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 8.51, 8.10 [12:56:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.52, 8.25, 7.52 [12:57:10] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:57:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 9.04, 8.26 [12:57:54] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.42, 1.67, 1.58 [12:58:18] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.29, 8.08, 7.54 [12:59:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.20, 8.66, 8.22 [13:00:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.22, 8.20, 8.12 [13:01:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.68, 8.30, 8.13 [13:06:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 8.70, 8.38 [13:08:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.15, 8.31, 8.27 [13:08:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.51, 8.57, 8.09 [13:10:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.41, 8.15, 7.99 [13:10:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 8.04, 7.87 [13:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:12:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.60, 8.08, 7.92 [13:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:15:49] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [13:15:49] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:19:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.98, 3.97, 3.25 [13:20:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:21:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.48, 3.16 [13:24:03] RECOVERY - wiki.newnormal.fans - reverse DNS on sslhost is OK: SSL OK - wiki.newnormal.fans reverse DNS resolves to cp21.miraheze.org - CNAME OK [13:26:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:27:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.79, 3.35 [13:28:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.50, 3.13 [13:29:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.63, 3.24 [13:30:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.10, 8.62, 8.22 [13:30:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 4.16, 3.42 [13:31:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.68, 3.42 [13:31:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.33, 8.50, 8.23 [13:33:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.87, 3.51 [13:33:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 4.07, 3.49 [13:37:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.79, 3.60 [13:37:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.92, 3.62 [13:38:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.91, 3.82, 3.68 [13:39:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.32, 4.34, 3.82 [13:39:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.83, 8.82, 8.21 [13:39:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.02, 9.40, 8.59 [13:40:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.00, 4.54, 3.98 [13:41:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.86, 3.70 [13:42:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.90, 3.81 [13:43:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 3.07, 3.33 [13:43:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.89, 7.58, 7.88 [13:43:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.43, 9.04, 8.75 [13:45:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 3.82, 3.68 [13:45:22] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [13:46:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:47:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.55, 3.59 [13:50:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.28, 2.70, 3.28 [13:51:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.67, 2.87, 3.29 [13:55:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:57:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.97, 3.66 [13:58:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.29, 8.63, 8.09 [13:58:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.58, 3.36 [13:58:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 8.62, 8.10 [14:00:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.41, 8.47, 8.10 [14:00:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.18, 3.13, 3.22 [14:00:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:01:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.22, 8.05, 8.40 [14:02:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.25, 7.97, 7.98 [14:03:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 3.01, 3.31 [14:16:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:19:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.68, 8.17, 7.97 [14:21:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.79, 3.40 [14:22:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.90, 8.98, 8.57 [14:23:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.29, 8.64, 8.26 [14:23:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 8.72, 8.20 [14:23:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.75, 3.37 [14:25:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.50, 3.41 [14:25:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.90, 8.40, 8.15 [14:27:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.41, 3.11, 3.27 [14:27:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.61, 4.03, 3.52 [14:28:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 9.47, 8.88 [14:30:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.36, 9.39, 8.92 [14:31:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.59, 3.44 [14:31:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.57, 7.77, 8.14 [14:31:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.63, 8.87, 8.41 [14:31:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.65, 3.47 [14:34:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.99, 10.04, 9.26 [14:34:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.89, 3.69, 3.27 [14:36:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.59, 9.38, 9.11 [14:36:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:36:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.93, 3.40, 3.22 [14:37:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 3.07, 3.30 [14:39:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 3.00, 3.28 [14:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:44:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.20, 7.79, 8.48 [14:44:28] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:49:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.95, 8.04, 8.40 [14:50:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.18, 8.33, 8.53 [14:56:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 8.24, 8.42 [14:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:01:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.53, 3.27 [15:04:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:06:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.42, 8.62, 8.43 [15:06:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.54, 8.61, 8.45 [15:06:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.56, 7.71, 7.35 [15:08:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.92, 7.82, 8.17 [15:08:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.74, 7.56, 7.36 [15:11:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.92, 3.58 [15:12:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.27, 9.75, 8.93 [15:12:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.47, 3.14 [15:13:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.75, 3.57 [15:13:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 9.59, 8.95 [15:14:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:15:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.92, 3.64 [15:18:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.64, 2.89, 3.04 [15:19:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.69, 3.63 [15:19:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:20:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.33, 8.83, 8.34 [15:21:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.49, 9.80, 9.25 [15:24:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.32, 8.37, 8.26 [15:25:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 2.94, 3.34 [15:25:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 9.75, 9.39 [15:29:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.47, 3.28 [15:29:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.23, 10.12, 9.59 [15:31:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.27, 3.23 [15:34:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.57, 9.58, 8.76 [15:34:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.74, 9.42, 8.59 [15:36:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.37, 9.44, 8.81 [15:37:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.93, 9.57, 9.64 [15:38:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.07, 9.59, 8.94 [15:40:24] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.61, 9.30, 8.73 [15:40:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.41, 8.66, 8.67 [15:41:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.87, 10.81, 10.10 [15:42:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.02, 8.82, 8.62 [15:44:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.49, 9.07, 8.79 [15:45:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.84, 3.49, 3.16 [15:46:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.46, 8.52, 8.63 [15:47:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.02, 9.04, 9.65 [15:50:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.01, 9.46, 8.91 [15:50:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.01, 2.83 [15:50:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.33, 8.81, 8.39 [15:51:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.57, 3.29 [15:51:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.10, 2.82 [15:52:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 9.46, 8.97 [15:52:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.39, 3.05, 2.86 [15:52:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.32, 9.26, 8.89 [15:55:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.13, 3.18, 2.94 [15:56:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.32, 8.42, 8.33 [15:57:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.23, 3.27 [15:58:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.28, 9.42, 9.09 [16:04:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.41, 3.04 [16:04:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.52, 10.06, 9.33 [16:05:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.77, 3.49 [16:06:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.30, 9.56, 9.03 [16:06:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.09, 3.18, 2.99 [16:07:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.43, 3.38 [16:08:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 9.27, 8.99 [16:09:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.88, 2.87, 3.18 [16:11:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.11, 4.16, 3.39 [16:11:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.96, 9.75, 9.13 [16:11:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.91, 1.88, 1.21 [16:13:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.91, 3.38 [16:13:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 9.06, 8.95 [16:13:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.58, 1.18 [16:14:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.32, 7.38, 8.27 [16:14:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.26, 9.79, 9.61 [16:15:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 3.39, 3.25 [16:17:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.10, 7.74, 8.40 [16:18:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.27, 2.87, 1.89 [16:19:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:19:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.23, 8.20, 7.78 [16:20:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 9.44, 9.46 [16:21:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.26, 4.05, 3.54 [16:22:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.66, 8.65, 9.16 [16:22:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.60, 1.66, 1.60 [16:22:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:23:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.82, 3.60, 3.44 [16:24:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.41, 3.18 [16:24:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.14, 9.20, 9.30 [16:25:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 3.14, 3.29 [16:25:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.67, 8.08, 7.51 [16:26:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.15, 3.12 [16:26:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.61, 9.09, 9.23 [16:27:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.06, 7.84, 7.48 [16:29:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.32, 8.16, 8.05 [16:29:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 8.84, 8.57 [16:32:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.29, 9.70, 9.48 [16:36:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.01, 9.56, 9.50 [16:36:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.67, 2.85, 1.90 [16:37:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.84, 9.55, 8.98 [16:39:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.35, 9.16, 8.92 [16:40:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.47, 1.85, 1.74 [16:41:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 9.72, 9.15 [16:42:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.53, 1.42, 1.59 [16:43:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.58, 8.96, 8.95 [16:46:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 9.23, 9.25 [16:47:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.73, 3.47 [16:48:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.59, 8.55, 8.99 [16:51:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.22, 8.85, 8.33 [16:51:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.73, 3.14, 3.29 [16:52:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.45, 9.30, 8.55 [16:53:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.81, 8.23, 8.50 [16:54:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.29, 9.43, 8.70 [16:56:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.01, 9.63, 8.86 [16:57:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.64, 8.77, 8.69 [16:58:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.64, 7.65, 8.34 [16:58:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.48, 8.95, 8.70 [17:01:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.96, 3.76, 3.50 [17:02:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.61, 8.70, 8.64 [17:05:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.57, 2.89, 3.22 [17:05:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.06, 8.75, 8.00 [17:05:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.29, 9.30, 8.97 [17:06:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.31, 9.27, 8.84 [17:07:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.42, 8.50, 8.02 [17:07:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:07:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 9.44, 9.06 [17:08:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.91, 9.29, 8.89 [17:09:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.84, 10.07, 9.32 [17:14:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:15:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.39, 3.88, 3.50 [17:16:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.41, 7.95, 8.39 [17:16:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.43, 2.96 [17:17:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.69, 9.78, 9.60 [17:18:41] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.91, 2.85, 2.81 [17:19:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.77, 3.29, 3.39 [17:19:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:20:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.35, 9.39, 8.88 [17:20:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.77, 7.87, 8.33 [17:21:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.07, 8.85, 8.74 [17:23:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.75, 3.56 [17:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:24:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.62, 8.68, 8.32 [17:25:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.48, 3.48 [17:25:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.25, 7.37, 8.18 [17:27:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.47, 4.59, 3.92 [17:27:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.47, 3.21 [17:28:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.52, 9.19, 8.56 [17:29:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.35, 8.54, 8.43 [17:29:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:29:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.09, 9.31, 9.27 [17:29:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.33, 3.19 [17:31:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.66, 3.70 [17:31:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.17, 7.86, 8.18 [17:31:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.77, 8.97, 9.15 [17:32:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.76, 8.95, 8.66 [17:32:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:33:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.64, 8.33, 8.36 [17:33:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.23, 9.50, 9.32 [17:34:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.15, 9.37, 8.84 [17:35:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.36, 8.91, 9.14 [17:36:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.64, 9.82, 9.10 [17:37:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:37:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.87, 7.88, 8.15 [17:37:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.59, 3.28 [17:38:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.25, 8.84, 8.53 [17:39:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.24, 2.98 [17:39:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.16, 3.26, 3.19 [17:40:53] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.37, 7.97, 8.25 [17:41:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.26, 3.26, 3.02 [17:41:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.33, 9.87, 9.42 [17:43:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.72, 8.95, 9.14 [17:44:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.78, 7.62, 8.48 [17:47:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:48:48] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [17:48:48] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:49:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.37, 7.68, 8.48 [17:50:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 7.83, 7.81 [17:52:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.34, 7.92, 7.84 [17:53:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.88, 3.65 [17:55:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.42, 3.50 [17:57:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 3.63, 3.19 [17:57:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.66, 8.99, 8.74 [17:59:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.77, 3.61 [17:59:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.60, 3.23 [18:01:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.32, 3.47 [18:01:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.50, 3.20 [18:01:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.84, 3.36 [18:03:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.06, 8.17, 8.50 [18:05:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.61, 3.06, 3.33 [18:05:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.12, 9.86, 8.78 [18:07:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 9.43, 8.75 [18:07:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.41, 3.41 [18:09:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.33, 3.19 [18:09:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.61, 3.46 [18:11:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.33, 8.32, 8.44 [18:13:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.42, 3.35 [18:13:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.18, 2.94, 3.08 [18:13:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.70, 3.55 [18:14:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 8.95, 8.66 [18:15:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.83, 3.50 [18:15:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.14, 8.94, 8.64 [18:17:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.62, 3.46 [18:17:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 3.15, 3.38 [18:18:21] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:18:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.15, 7.67, 8.23 [18:19:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 3.13, 3.30 [18:21:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.82, 8.16, 8.48 [18:22:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:23:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.59, 3.43 [18:23:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.42, 3.18 [18:23:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.12, 3.47, 3.34 [18:25:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.11, 3.27 [18:25:12] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 2.97, 3.04 [18:25:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 2.93, 3.15 [18:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:27:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 8.90, 8.69 [18:33:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.98, 7.73, 8.28 [18:37:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.22, 3.07 [18:39:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.09, 8.96, 8.57 [18:41:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.89, 2.72, 2.91 [18:41:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.41, 8.65, 8.53 [18:43:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.96, 8.01, 8.30 [18:44:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 8.66, 8.04 [18:46:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.63, 8.18, 7.93 [18:53:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.90, 4.30, 3.39 [18:53:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.02, 4.58, 3.63 [18:53:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.72, 3.08 [18:53:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.88, 8.36, 8.25 [18:57:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.80, 7.87, 8.09 [19:02:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.48, 8.57, 7.96 [19:08:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.01, 8.45, 8.14 [19:11:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.00, 9.20, 8.59 [19:13:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.22, 8.41, 8.37 [19:19:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.66, 9.42, 8.51 [19:21:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.55, 9.08, 8.50 [19:25:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.42, 7.91, 6.99 [19:25:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.02, 7.13, 7.90 [19:25:46] PROBLEM - mw121 Disk Space on mw121 is CRITICAL: DISK CRITICAL - free space: / 150 MB (0% inode=83%); [19:26:17] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:22] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:45] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:47] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:55] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.47, 6.29, 4.71 [19:26:56] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:00] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:46] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:46] RECOVERY - mw121 Disk Space on mw121 is OK: DISK OK - free space: / 5379 MB (24% inode=83%); [19:27:53] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:53] PROBLEM - mw102 Disk Space on mw102 is WARNING: DISK WARNING - free space: / 1667 MB (7% inode=83%); [19:27:55] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.008 second response time [19:27:55] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.242 second response time [19:28:00] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:08] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:12] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:34] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:40] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:03] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:05] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:09] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:11] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:13] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.02, 7.40, 7.03 [19:30:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.05, 6.22, 5.15 [19:31:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.74, 8.39, 7.95 [19:32:03] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.540 second response time [19:32:30] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.583 second response time [19:32:37] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 6.356 second response time [19:33:15] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.746 second response time [19:33:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.67, 8.97, 8.23 [19:33:20] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.186 second response time [19:33:58] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.262 second response time [19:34:06] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.925 second response time [19:34:40] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 10.02, 6.40, 4.33 [19:34:45] PROBLEM - mw111 Disk Space on mw111 is CRITICAL: DISK CRITICAL - free space: / 336 MB (1% inode=83%); [19:34:55] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.141 second response time [19:35:09] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.14, 8.23, 8.05 [19:35:46] PROBLEM - mw121 Disk Space on mw121 is CRITICAL: DISK CRITICAL - free space: / 1285 MB (5% inode=83%); [19:36:11] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.497 second response time [19:36:21] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:27] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.106 second response time [19:36:27] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 8.646 second response time [19:36:36] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.970 second response time [19:36:37] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.49, 6.36, 4.60 [19:36:56] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.883 second response time [19:37:03] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.123 second response time [19:37:11] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.585 second response time [19:37:21] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.547 second response time [19:37:31] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [19:39:11] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.134 second response time [19:39:29] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 9.654 second response time [19:39:31] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.981 second response time [19:43:46] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.314 second response time [19:44:27] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.020 second response time [19:47:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.27, 9.33, 7.50 [19:50:25] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.98, 6.54, 4.02 [19:51:59] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:22] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:36] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:39] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:42] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:27] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.036 second response time [19:54:39] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.373 second response time [19:54:44] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.032 second response time [19:54:45] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.226 second response time [19:56:03] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 5.904 second response time [20:00:22] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:25] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 6.58, 7.92, 6.31 [20:00:56] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:06] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:11] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:16] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:20] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 4.182 second response time [20:02:25] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.69, 6.78, 6.09 [20:02:56] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.750 second response time [20:03:04] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.909 second response time [20:03:11] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.890 second response time [20:03:17] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.841 second response time [20:05:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.01, 9.40, 8.31 [20:06:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.84, 9.14, 8.20 [20:07:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 9.31, 8.42 [20:07:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.21, 9.46, 9.31 [20:08:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.46, 8.29, 8.02 [20:10:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.81, 10.04, 8.72 [20:12:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 9.92, 8.84 [20:14:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.02, 8.42, 8.44 [20:17:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.51, 9.18, 9.12 [20:18:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.83, 8.86, 8.57 [20:19:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.51, 9.39, 9.22 [20:20:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.58, 9.69, 8.90 [20:21:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.06, 10.05, 9.21 [20:22:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 9.57, 8.96 [20:23:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.48, 9.95, 9.29 [20:25:57] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:28] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 9.90, 9.39 [20:27:59] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.563 second response time [20:29:25] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.200 second response time [20:29:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.18, 8.00, 8.49 [20:29:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 9.75, 9.42 [20:30:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.71, 7.61, 8.38 [20:31:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.13, 8.80, 8.28 [20:33:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.25, 8.37, 8.17 [20:34:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.10, 9.08, 8.80 [20:36:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.21, 8.61, 8.65 [20:37:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.76, 9.53, 9.30 [20:39:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.32, 9.19, 9.21 [20:40:12] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 1.41, 1.01 [20:40:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.15, 8.05, 8.43 [20:42:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.28, 1.01 [20:50:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.72, 8.89, 8.48 [20:53:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.54, 9.32, 9.17 [20:54:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.96, 8.70, 8.11 [20:55:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.56, 9.53, 9.28 [20:56:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.71, 9.15, 8.33 [20:58:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.11, 8.99, 8.36 [20:59:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.32, 10.13, 9.55 [20:59:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.84, 8.32, 8.50 [21:01:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.01, 9.11, 8.68 [21:03:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.15, 9.68, 9.00 [21:03:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.92, 9.95, 9.03 [21:05:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.65, 9.66, 9.08 [21:06:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.42, 9.84, 9.01 [21:08:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.72, 9.30, 8.91 [21:12:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.19, 9.52, 9.06 [21:13:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.35, 9.66, 9.43 [21:14:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.45, 8.70, 8.80 [21:15:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.49, 7.67, 8.40 [21:15:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.84, 9.31, 9.85 [21:18:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.81, 7.98, 8.49 [21:19:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.02, 2.82, 3.94 [21:21:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.55, 3.78, 4.15 [21:23:53] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:54] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.199 second response time [21:27:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 8.68, 8.92 [21:28:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.76, 8.88, 8.80 [21:30:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.15, 8.49, 8.66 [21:31:12] !log macfan@mw102:/$ sudo -u www-data rm -rf /tmp/* [21:31:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.62, 8.91, 9.03 [21:31:53] RECOVERY - mw102 Disk Space on mw102 is OK: DISK OK - free space: / 14159 MB (64% inode=83%); [21:32:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 7.77, 8.37 [21:33:04] !log macfan@mw121:/$ sudo -u www-data rm -rf /tmp/* [21:33:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:46] RECOVERY - mw121 Disk Space on mw121 is OK: DISK OK - free space: / 14127 MB (63% inode=83%); [21:33:59] !log macfan@mw111:/$ sudo -u www-data rm -rf /tmp/* [21:34:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:45] RECOVERY - mw111 Disk Space on mw111 is OK: DISK OK - free space: / 14136 MB (64% inode=83%); [21:36:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 8.35, 7.96 [21:36:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.01, 8.69, 8.65 [21:36:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 8.21, 7.98 [21:37:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.74, 9.33 [21:38:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.93, 8.14, 7.93 [21:38:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.81, 9.49, 8.95 [21:38:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.69, 8.06, 7.96 [21:39:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.96, 9.73, 9.37 [21:41:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.61, 3.99 [21:41:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.82, 11.33, 10.01 [21:42:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.84, 8.54, 8.17 [21:45:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.29, 3.19, 3.99 [21:45:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.63, 9.89, 9.77 [21:45:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.52, 8.21, 8.13 [21:46:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.42, 9.37, 9.22 [21:49:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.31, 3.98 [21:49:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.21, 8.80, 8.37 [21:51:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.56, 8.05, 8.15 [21:53:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.02, 2.57, 3.39 [21:55:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.95, 2.90, 3.29 [21:57:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:59:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.46, 3.54 [21:59:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.98, 8.45, 8.24 [22:00:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:01:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.64, 3.71 [22:01:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.62, 8.15, 8.16 [22:02:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.28, 7.88, 8.40 [22:03:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.55, 3.53 [22:05:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.71, 3.73 [22:05:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.12, 3.36 [22:05:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.45, 8.46, 8.78 [22:06:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.78, 9.00, 8.75 [22:06:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.82, 6.08, 4.92 [22:07:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.27, 8.67, 8.82 [22:08:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.93, 6.10, 5.07 [22:11:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.72, 3.67 [22:12:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.92, 9.20, 8.87 [22:13:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.47, 3.36 [22:14:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.29, 9.00, 8.85 [22:15:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 4.02, 3.58 [22:15:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.04, 9.21, 8.97 [22:17:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.66, 3.49 [22:17:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.56, 9.00, 8.93 [22:18:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.07, 8.06, 8.49 [22:20:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.81, 9.16, 8.51 [22:20:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:21:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.65, 3.50 [22:21:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.99, 9.93, 9.32 [22:22:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.28, 8.38, 8.30 [22:22:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 8.63, 8.63 [22:23:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.70, 9.69, 9.30 [22:24:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.07, 7.81, 8.34 [22:25:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.71, 4.00, 3.56 [22:27:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.84, 3.55 [22:27:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.58, 3.56 [22:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:31:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.60, 4.02, 3.69 [22:31:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 9.02, 8.99 [22:33:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.99, 8.79, 8.92 [22:34:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.99, 8.85, 8.06 [22:35:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 4.41, 3.96 [22:36:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.13, 8.58, 8.40 [22:37:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.93, 9.68, 9.24 [22:37:55] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.18, 1.64, 1.03 [22:39:50] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.30, 0.97 [22:40:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.94, 3.87 [22:42:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.12, 7.79, 7.99 [22:42:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.80, 8.12, 8.35 [22:43:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.44, 3.46, 3.75 [22:48:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.57, 9.26, 8.53 [22:49:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.11, 9.75, 9.77 [22:50:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.64, 9.10, 8.72 [22:50:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.84, 9.46, 8.69 [22:51:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.56, 8.54, 7.85 [22:52:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.17, 9.64, 8.84 [22:52:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.70, 3.02, 3.38 [22:53:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.10, 7.96, 7.27 [22:54:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.34, 9.14, 8.75 [22:55:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.14, 7.62, 7.23 [22:55:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.90, 3.72 [22:55:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.49, 8.49, 7.99 [22:56:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.88, 3.44, 3.50 [22:59:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.43, 3.58 [22:59:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.04, 8.64, 8.21 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.81, 9.77, 9.14 [23:00:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.32, 2.99, 3.33 [23:01:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.69, 3.65 [23:01:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.03, 8.50, 8.21 [23:02:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.09, 9.18, 9.00 [23:03:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.99, 3.74, 3.67 [23:04:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.54, 4.46, 3.82 [23:05:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 4.00, 3.78 [23:06:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.80, 10.15, 9.39 [23:06:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.89, 3.69 [23:07:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.58, 3.65 [23:08:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.41, 9.72, 9.32 [23:08:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.46, 7.92, 8.46 [23:08:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.84, 3.67 [23:09:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.89, 3.76 [23:10:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.56, 3.58 [23:11:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.91, 3.78 [23:12:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.02, 9.46, 9.27 [23:13:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.06, 9.29, 8.93 [23:14:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.61, 8.28, 8.86 [23:14:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.28, 8.85, 8.66 [23:14:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 3.72, 3.61 [23:15:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.29, 4.23, 3.89 [23:15:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 9.15, 8.91 [23:16:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.32, 9.06, 8.77 [23:18:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.95, 3.75 [23:19:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.75, 3.78 [23:20:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 1.45, 0.83 [23:21:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 4.11, 3.92 [23:22:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.24, 1.37, 0.88 [23:23:26] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1404s [23:23:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.41, 3.68 [23:23:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.57, 8.06, 8.44 [23:23:54] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.35, 1.99, 1.40 [23:25:54] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.89, 1.56, 1.31 [23:26:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.40, 7.56, 8.28 [23:27:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 3.60, 3.66 [23:28:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.98, 7.91, 8.48 [23:28:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.91, 3.99, 3.75 [23:28:48] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1724s [23:28:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.50, 3.60 [23:32:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.10, 2.71, 3.30 [23:33:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.03, 2.97, 3.98 [23:34:58] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.26, 6.68, 5.59 [23:35:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.21, 3.79, 4.15 [23:36:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.70, 6.03, 5.49 [23:40:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 4.05, 3.63 [23:42:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.59, 3.51 [23:45:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:50:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.95, 3.99, 3.74 [23:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:54:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.78, 3.74