[00:01:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.25, 9.84, 9.71 [00:02:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.22, 9.86, 9.58 [00:02:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.10, 8.73, 8.93 [00:02:37] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.00, 9.80, 9.73 [00:04:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.85, 8.85, 9.23 [00:04:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:06:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.38, 9.82, 9.40 [00:10:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.84, 8.03, 8.37 [00:11:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [00:11:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.18, 8.90, 9.28 [00:11:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.83, 9.95, 9.60 [00:15:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.36, 9.39, 9.47 [00:15:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.83, 9.44, 9.50 [00:16:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.33, 9.21, 8.72 [00:17:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.29, 9.68, 9.56 [00:18:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.84, 9.38, 8.84 [00:19:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.23, 9.63, 9.57 [00:20:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.95, 10.01, 9.36 [00:21:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.23, 10.10, 9.16 [00:22:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.84, 9.55, 9.27 [00:22:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.17, 9.50, 9.00 [00:23:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.12, 9.33, 8.98 [00:24:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.10, 9.04, 8.91 [00:24:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.01, 9.63, 9.19 [00:25:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.22, 9.93, 9.71 [00:26:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 9.10, 9.03 [00:27:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.70, 8.86, 9.34 [00:27:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.49, 2.73 [00:27:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 3.49, 2.97 [00:29:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.21, 2.72 [00:31:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.09, 7.38, 8.41 [00:32:37] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.55, 3.16 [00:34:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.55, 9.68, 9.21 [00:34:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.70, 7.60, 8.35 [00:35:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.96, 8.23, 8.52 [00:36:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.59, 9.09, 8.84 [00:36:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.76, 9.31, 9.14 [00:37:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.75, 3.32 [00:37:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.55, 7.13, 8.08 [00:38:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.76, 7.37, 8.23 [00:38:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.28, 7.31, 8.47 [00:38:43] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:39:10] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.242 second response time [00:39:47] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:07] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.00, 7.26, 8.31 [00:40:32] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:55] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [00:41:04] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:31] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22220 bytes in 0.552 second response time [00:42:42] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.336 second response time [00:42:50] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.020 second response time [00:43:02] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [00:43:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.59, 3.58, 3.44 [00:43:17] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.312 second response time [00:43:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.46, 7.24, 8.32 [00:43:30] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [00:44:45] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:56] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:02] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.57, 4.72, 3.87 [00:45:30] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [00:45:58] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22220 bytes in 3.970 second response time [00:46:03] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.783 second response time [00:46:42] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.203 second response time [00:46:53] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.335 second response time [00:46:58] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.325 second response time [00:49:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.73, 7.39, 7.19 [00:50:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.94, 8.57, 8.29 [00:51:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.96, 8.93, 8.54 [00:51:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 8.74, 8.08 [00:51:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.01, 8.19, 7.50 [00:52:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.83, 9.63, 8.71 [00:53:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.35, 8.74, 8.54 [00:53:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.70, 8.76, 8.18 [00:53:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.71, 8.82, 7.83 [00:54:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.17, 10.22, 8.89 [00:54:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 9.58, 8.81 [00:54:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.61, 9.43, 8.38 [00:55:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.23, 8.20, 7.72 [00:56:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.51, 9.38, 8.76 [00:56:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.89, 9.32, 8.45 [00:57:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.25, 9.55, 8.91 [00:57:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.02, 9.77, 8.70 [00:58:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.11, 10.00, 9.05 [00:58:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.04, 10.35, 9.26 [00:58:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.05, 9.70, 8.68 [00:58:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.84, 3.96 [00:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:59:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.58, 10.01, 8.51 [01:00:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.13, 9.48, 8.73 [01:01:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.93, 9.48, 8.49 [01:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:02:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 3.89, 3.92 [01:04:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.21, 10.13, 9.15 [01:04:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.60, 3.82 [01:05:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.78, 9.28, 9.18 [01:05:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.11, 10.40, 9.08 [01:05:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.77, 9.48, 9.14 [01:06:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.09, 9.46, 9.31 [01:06:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.73, 3.96, 3.92 [01:07:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.73, 9.07, 8.75 [01:07:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.63, 10.15, 9.41 [01:08:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.17, 3.32, 3.69 [01:09:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.11, 9.79, 9.42 [01:09:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.50, 8.18, 8.46 [01:10:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.50, 9.98, 9.51 [01:10:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.07, 9.65, 9.34 [01:10:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.65, 3.77 [01:12:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.20, 9.82, 9.52 [01:13:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.22, 9.95, 9.62 [01:17:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.87, 9.83, 9.71 [01:18:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.39, 10.35, 9.65 [01:19:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.59, 10.50, 9.89 [01:20:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.75, 3.97 [01:22:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 9.56, 9.97 [01:23:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.76, 9.29, 8.73 [01:23:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.17, 9.30, 9.56 [01:24:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.55, 3.78 [01:25:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.92, 8.39, 8.47 [01:26:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.70, 9.89, 9.94 [01:27:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.77, 9.47, 9.53 [01:29:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.86, 9.38, 9.50 [01:30:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.69, 9.77, 9.23 [01:30:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.69, 3.82 [01:31:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.24, 9.66, 9.53 [01:31:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.97, 1.84, 1.53 [01:33:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.43, 9.23, 9.39 [01:34:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.86, 9.74, 9.38 [01:34:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 3.97, 3.90 [01:35:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.48, 3.03 [01:35:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.35, 1.39 [01:36:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.90, 9.39, 9.51 [01:36:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.69, 3.79 [01:37:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.85, 2.88, 2.86 [01:39:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.87, 9.41, 9.35 [01:40:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.86, 9.30, 9.47 [01:41:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.44, 9.21, 9.15 [01:43:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.65, 9.86, 9.64 [01:43:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.26, 8.75, 8.98 [01:44:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 2.89, 3.36 [01:47:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.70, 9.40, 9.45 [01:47:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:48:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.06, 10.08, 9.63 [01:51:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.43, 1.55, 1.13 [01:51:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.10, 9.61, 9.56 [01:52:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:52:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.31, 8.67, 8.11 [01:53:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.70, 1.24, 1.07 [01:54:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.30, 7.69, 8.47 [01:54:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.38, 8.43, 8.09 [01:55:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.39, 7.82, 8.38 [01:56:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.76, 9.62, 9.74 [01:59:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.92, 2.21, 1.02 [01:59:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 8.62, 8.59 [02:00:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.29, 9.59, 9.66 [02:02:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.91, 8.69, 8.60 [02:02:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.50, 9.21, 9.54 [02:03:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.06, 7.59, 8.39 [02:04:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.45, 9.50, 8.92 [02:05:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.28, 9.62, 8.68 [02:06:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 9.18, 8.87 [02:07:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.09, 9.27, 8.65 [02:07:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.33, 8.10, 8.42 [02:10:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.43, 8.85, 9.14 [02:10:53] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [02:11:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.64, 9.03, 8.60 [02:11:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.66, 8.43, 8.53 [02:12:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.06, 9.03, 9.19 [02:12:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.36, 3.11 [02:14:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.73, 2.79, 2.93 [02:15:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.29, 9.31, 8.89 [02:15:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.40, 8.36, 8.50 [02:15:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.61, 8.28, 8.48 [02:16:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.74, 7.73, 8.36 [02:16:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.24, 9.52, 9.33 [02:19:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.54, 8.39, 8.51 [02:21:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.69, 9.17, 8.81 [02:21:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.62, 9.87, 9.05 [02:22:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.04, 9.28, 8.77 [02:22:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.10, 9.02, 8.81 [02:23:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.25, 9.38, 8.98 [02:23:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.64, 9.85, 9.12 [02:23:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.24, 9.38, 8.99 [02:24:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.47, 8.56, 8.68 [02:25:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.23, 8.96, 8.88 [02:26:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.30, 9.88, 9.20 [02:28:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.36, 10.00, 9.32 [02:29:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.71, 9.86, 9.49 [02:30:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.97, 9.81, 9.32 [02:32:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.69, 10.02, 9.45 [02:34:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.34, 9.14, 9.21 [02:34:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.80, 9.54, 9.07 [02:34:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.10, 9.37, 9.88 [02:37:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.42, 10.27, 9.42 [02:38:53] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [02:40:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.02, 9.56, 9.32 [02:40:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.96, 9.12, 9.03 [02:42:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.24, 9.49, 9.34 [02:42:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:43:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.45, 9.69, 9.50 [02:43:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.02, 9.10, 8.92 [02:44:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.50, 9.58, 9.36 [02:44:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.38, 9.03, 9.36 [02:45:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.49, 8.94, 8.89 [02:46:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.21, 8.89, 9.13 [02:46:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.70, 8.47, 9.11 [02:46:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.37, 2.97 [02:48:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 3.03, 2.89 [02:48:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.09, 7.46, 8.37 [02:50:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.71, 9.67, 9.41 [02:51:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.45, 10.02, 9.70 [02:52:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.30, 9.30, 9.29 [02:53:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.22, 9.52, 9.57 [02:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:53:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.06, 9.11, 8.91 [02:55:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.48, 9.16, 8.95 [02:56:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.59, 8.56, 8.50 [02:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:00:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.17, 8.11, 8.35 [03:02:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.20, 8.96, 8.79 [03:04:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.71, 8.00, 8.45 [03:04:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.48, 8.07, 8.50 [03:04:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:05:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.75, 9.42, 9.19 [03:05:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.97, 7.83, 8.35 [03:07:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.37, 9.07, 9.09 [03:09:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:13:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.63, 8.90 [03:14:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.37, 8.60, 8.47 [03:15:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.03, 9.35, 9.12 [03:15:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 8.44, 8.16 [03:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:15:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.10, 8.87, 8.72 [03:16:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.94, 8.96, 8.60 [03:18:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.60, 9.59, 8.91 [03:18:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.84, 9.09, 8.87 [03:19:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.91, 9.39, 9.20 [03:19:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.34, 2.80 [03:19:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.97, 9.35, 8.56 [03:20:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.49, 8.99, 8.79 [03:20:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.08, 8.50, 8.48 [03:21:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.43, 9.16, 8.57 [03:22:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.73, 7.54, 8.28 [03:22:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.49, 9.20, 9.05 [03:23:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.59, 3.04 [03:23:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.43, 7.48, 8.41 [03:23:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.02, 8.18, 8.27 [03:23:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.17, 9.03, 8.79 [03:24:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 9.33, 9.10 [03:25:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.37, 2.81, 2.82 [03:25:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.13, 8.63, 8.67 [03:26:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.08, 9.03, 9.04 [03:27:18] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.05, 1.46, 1.95 [03:28:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.50, 8.64, 8.57 [03:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:32:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.87, 7.85, 8.49 [03:33:18] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.01, 1.18, 1.66 [03:35:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.84, 7.49, 8.25 [03:36:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.84, 8.23, 8.50 [03:40:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.20, 2.87 [03:42:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.17, 3.09, 2.87 [03:43:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.03, 8.52, 8.38 [03:44:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.60, 9.54, 8.89 [03:44:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.73, 8.89, 8.27 [03:45:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.74, 9.11, 8.52 [03:45:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.64, 8.16, 8.26 [03:45:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 8.33, 8.26 [03:46:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.36, 9.31, 8.87 [03:46:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.39, 9.58, 8.76 [03:47:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.23, 8.88, 8.23 [03:48:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.25, 9.33, 8.55 [03:48:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.02, 9.15, 8.69 [03:48:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.70, 9.30, 8.67 [03:49:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.74, 8.15, 8.04 [03:49:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.00, 8.42, 8.34 [03:50:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.71, 8.51, 8.35 [03:50:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.37, 8.50, 8.45 [03:52:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.10, 8.41, 8.32 [03:52:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.85, 8.40, 8.50 [03:59:02] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [04:00:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.42, 8.00, 8.45 [04:05:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.23, 8.97, 8.46 [04:07:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.09, 1.77, 1.52 [04:07:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.29, 8.34, 8.29 [04:07:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.38, 2.87 [04:09:18] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.31, 1.62, 1.50 [04:10:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 8.39, 8.09 [04:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:12:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.86, 9.54, 8.69 [04:13:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.92, 8.73, 8.54 [04:14:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.06, 8.73, 8.49 [04:15:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.24, 3.07 [04:16:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.24, 8.37, 8.16 [04:16:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.32, 8.08, 8.27 [04:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:17:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 8.18, 8.39 [04:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:24:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.54, 9.18, 8.65 [04:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:26:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.55, 8.45, 8.43 [04:31:37] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:49:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:54:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:54:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 8.38, 8.10 [04:57:37] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:59:07] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [04:59:07] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4066102504730225 seconds [05:00:25] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.10, 8.12, 8.16 [05:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:15:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.47, 1.80, 1.21 [05:17:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.56, 1.37, 1.12 [05:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:04:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.05, 8.81, 7.90 [06:05:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:06:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.88, 8.58, 7.94 [06:08:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.07, 8.93, 8.13 [06:10:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.96, 8.71, 8.13 [06:10:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 8.48, 7.75 [06:11:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.20, 8.70, 8.09 [06:14:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.88, 8.43, 7.93 [06:17:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.66, 9.50, 8.58 [06:18:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.33, 8.12, 8.16 [06:19:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:20:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 8.43, 8.02 [06:21:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.99, 9.26, 8.72 [06:22:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 7.65, 7.29 [06:24:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.80, 7.70, 7.36 [06:25:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.41, 8.56, 7.86 [06:27:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.90, 8.03, 8.38 [06:27:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.15, 7.90, 7.70 [06:28:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.41, 9.46, 8.67 [06:29:13] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [06:30:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.89, 8.83, 8.54 [06:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:34:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.00, 8.12, 8.34 [06:35:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.23, 8.79, 8.62 [06:37:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.51, 9.16, 8.47 [06:38:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.57, 8.30, 7.80 [06:39:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.56, 8.53, 8.32 [06:40:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.96, 8.85, 8.55 [06:40:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.12, 7.98, 7.73 [06:41:11] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/4b99fbea16b4...9807d6079917 [06:41:13] [02miraheze/dns] 07Reception123 039807d60 - add arrieretoile.fr zone [06:41:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.18, 9.48, 8.92 [06:43:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.72, 8.51, 8.65 [06:43:35] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.99, 8.21, 8.23 [06:44:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.63, 8.27, 8.40 [06:45:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.36, 7.70, 8.33 [06:47:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.46, 2.00, 1.36 [06:47:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:48:50] [02miraheze/ssl] 07Reception123 pushed 034 commits to 03master [+4/-0/±4] 13https://github.com/miraheze/ssl/compare/a70a4fd93d47...48707874d1fa [06:48:52] [02miraheze/ssl] 07Reception123 035b46f67 - add dnd.bellinrattin.it [06:48:53] [02miraheze/ssl] 07Reception123 039281aae - add arrieretoile.fr cert [06:48:55] [02miraheze/ssl] 07Reception123 03c13e593 - readd storytime.jdstroy.cf cert [06:48:56] [02miraheze/ssl] 07Reception123 034870787 - add wiki.sketch.town cert [06:49:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.77, 1.57, 1.28 [06:49:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.50, 8.81, 8.64 [06:49:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.26, 9.01, 8.57 [06:50:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:51:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.29, 9.70, 8.98 [06:53:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.43, 8.27, 8.37 [06:53:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.74, 8.99, 8.81 [06:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:00:57] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [07:00:57] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40639066696167 seconds [07:04:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.08, 8.09, 8.42 [07:06:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.37, 9.79, 8.82 [07:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:08:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.52, 8.83, 8.64 [07:10:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.29, 9.25, 8.78 [07:12:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.85, 9.13, 8.81 [07:12:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 8.62, 8.25 [07:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:18:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.76, 9.54, 9.36 [07:19:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 2.87, 2.08 [07:19:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.08, 2.29 [07:21:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 2.56, 2.05 [07:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:21:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 2.89, 2.31 [07:22:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.59, 8.47, 8.45 [07:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:29:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 3.35, 2.49 [07:29:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.09, 8.18, 7.73 [07:29:44] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.38, 9.61, 8.54 [07:29:51] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [07:30:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.79, 2.99, 2.47 [07:31:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.33, 8.21, 7.79 [07:31:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.85, 9.52, 8.65 [07:32:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.79, 7.78, 8.49 [07:32:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.35, 9.09, 8.67 [07:33:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.30, 9.06, 8.73 [07:35:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.44, 8.19, 8.31 [07:36:01]