[00:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.98, 3.94 [00:01:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.79, 2.33, 1.61 [00:02:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.32, 5.02, 4.33 [00:03:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.94, 1.93 [00:06:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.19, 1.75, 1.63 [00:08:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.44, 1.28, 1.47 [00:20:00] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.73, 1.53 [00:20:38] RECOVERY - wiki.dereric.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dereric.gq' will expire on Sat 22 Oct 2022 22:30:02 GMT +0000. [00:21:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.66, 1.65, 1.52 [00:25:33] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.05, 3.46, 2.73 [00:27:31] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.51, 3.39, 2.78 [00:29:28] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 1.57, 2.77, 2.63 [00:36:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.58, 9.81, 9.15 [00:40:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.80, 10.15, 9.44 [00:52:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.09, 3.96 [00:54:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 3.96, 4.19 [00:57:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 3.57, 3.95 [00:59:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 4.22, 4.15 [01:04:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.69, 3.96 [01:06:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.81, 4.00 [01:07:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.30, 1.64, 1.94 [01:10:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.82, 3.94 [01:12:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.73, 3.79 [01:14:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.64, 3.75 [01:15:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.39, 3.59, 3.91 [01:17:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 1.76, 1.79 [01:18:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.87, 3.96 [01:20:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.74, 3.89 [01:21:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.92, 4.04, 3.92 [01:23:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.86, 1.86 [01:23:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.81, 3.86 [01:25:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.89, 1.87 [01:26:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.19, 3.29, 3.75 [01:27:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.44, 1.72, 1.81 [01:27:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.44, 4.29, 4.01 [01:30:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.58, 3.62 [01:30:13] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 6.16, 3.23, 2.00 [01:31:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.87, 3.94 [01:32:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.38, 3.54 [01:34:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.80, 1.98, 1.78 [01:36:10] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.66, 1.68 [01:36:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 2.81, 3.30 [01:37:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.04, 4.10, 3.95 [01:38:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 3.53, 3.52 [01:39:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.91, 3.91 [01:40:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.56, 1.89, 1.80 [01:40:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.42, 3.49 [01:40:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.57, 3.50 [01:41:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 4.12, 3.99 [01:42:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 4.14, 3.75 [01:42:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.54, 3.51 [01:42:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.58, 9.76, 9.39 [01:43:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.84, 1.79 [01:43:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 4.00, 3.96 [01:44:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.50, 1.66 [01:44:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.78, 3.67 [01:44:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.86, 3.63 [01:44:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.81, 9.57, 9.38 [01:45:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.82, 1.79 [01:47:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.98, 2.17, 1.92 [01:47:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.66, 3.80 [01:48:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.77, 3.65 [01:50:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 11.08, 10.08 [01:52:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.72, 3.64 [01:52:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 11.10, 10.21 [01:53:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.45, 3.68 [01:54:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.88, 4.01, 3.70 [01:54:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.31, 11.61, 10.51 [01:55:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.80, 4.73, 4.12 [01:55:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.12, 1.87, 1.63 [01:56:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.67, 11.71, 10.69 [01:58:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.42, 12.20, 10.98 [01:59:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.38, 1.65, 1.60 [02:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.93, 3.86 [02:00:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.65, 3.74 [02:02:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.70, 3.74 [02:03:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.32, 1.91, 1.74 [02:04:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.60, 3.50, 3.68 [02:05:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.76, 3.97 [02:06:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.82, 3.82 [02:07:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.90, 1.51, 1.63 [02:08:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.56, 3.73 [02:08:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.86, 11.21, 11.23 [02:14:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.56, 3.58 [02:15:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.18, 3.48 [02:16:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.62, 4.09, 3.75 [02:18:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.72, 3.65 [02:21:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.54, 2.01, 1.38 [02:21:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.99, 3.80 [02:22:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.04, 11.18, 10.81 [02:23:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.79, 1.37 [02:23:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.98, 4.02, 3.84 [02:24:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 11.35, 10.93 [02:25:12] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 2.18, 1.58 [02:26:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 3.77, 3.62 [02:29:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.48, 1.84, 1.58 [02:30:58] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.58, 1.44, 1.46 [02:34:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.91, 3.74, 3.79 [02:34:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.64, 11.20, 10.88 [02:36:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.60, 4.01, 3.90 [02:36:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.95, 11.09, 10.89 [02:37:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.93, 3.32, 3.77 [02:38:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.47, 3.71 [02:38:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.36, 3.45, 3.95 [02:40:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 3.91, 4.04 [02:41:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.71, 3.82, 3.84 [02:42:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.60, 3.68 [02:43:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.53, 1.68, 1.98 [02:47:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.58, 1.91, 1.99 [02:47:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.66, 3.73, 3.86 [02:48:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.68, 3.74 [02:54:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.71, 1.67, 1.40 [02:56:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.75, 3.99 [02:56:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.93, 1.44, 1.35 [02:57:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.50, 1.76, 1.96 [02:58:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 11.54, 11.00 [03:00:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.48, 4.21, 4.08 [03:00:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 10.86, 10.80 [03:01:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 3.68, 3.61 [03:02:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.89, 3.98 [03:04:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 4.00, 4.00 [03:05:38] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.39, 1.68 [03:05:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.95, 3.98, 3.74 [03:06:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.80, 3.93 [03:08:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 4.22, 3.85 [03:08:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 4.29, 4.11 [03:10:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.96, 3.81 [03:10:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.26, 3.74, 3.95 [03:12:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.47, 8.67, 9.93 [03:15:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.69, 1.72 [03:15:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.14, 2.88, 3.35 [03:16:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.90, 3.93 [03:17:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 1.91, 1.80 [03:20:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.65, 3.86 [03:22:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.22, 3.04, 3.39 [03:22:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.53, 4.28, 4.05 [03:24:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.96, 3.96 [03:24:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.66, 4.38, 3.75 [03:25:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.97, 1.94 [03:26:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.32, 4.10, 3.72 [03:26:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.24, 4.42, 4.13 [03:27:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.96, 1.93 [03:29:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.88, 1.91 [03:33:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.63, 2.09, 1.97 [03:46:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.47, 2.05, 1.56 [03:52:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 10.71, 9.94 [03:52:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.91, 1.71 [03:56:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.57, 2.30, 1.77 [03:56:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.11, 10.88, 10.20 [03:57:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.91, 1.68 [03:58:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.26, 9.87, 9.91 [03:58:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.26, 1.48, 1.59 [04:01:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.56, 1.60 [04:04:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 10.28, 10.03 [04:06:11] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.99, 10.20, 10.04 [04:20:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.91, 3.12, 3.93 [04:24:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 3.90, 4.05 [04:26:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.82, 4.00 [04:28:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.58, 4.11, 4.11 [04:30:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.75, 3.98 [04:31:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.37, 1.63, 1.98 [04:32:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.71, 4.19, 4.12 [04:34:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.74, 3.97 [04:34:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.51, 3.95 [04:34:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.25, 9.72, 8.81 [04:35:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.66, 1.90 [04:35:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.55, 10.37, 9.70 [04:36:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.63, 9.42, 8.80 [04:37:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.42, 1.48, 1.80 [04:37:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.71, 9.31, 9.39 [04:41:38] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.17, 1.39, 1.68 [04:42:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.47, 3.43, 3.89 [04:46:12] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.86, 3.96, 4.02 [04:46:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 4.36, 4.07 [04:48:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.71, 3.94 [04:49:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.50, 3.36, 2.12 [04:49:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.91, 2.08, 1.84 [04:51:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.48, 10.18, 9.20 [04:53:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.29, 10.34, 9.40 [04:54:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.98, 4.55, 4.17 [04:54:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 4.07, 3.90 [04:55:19] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.04, 9.61, 9.23 [04:59:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.17, 1.81, 1.90 [04:59:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.95, 1.94 [05:02:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.63, 10.66, 9.86 [05:02:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.65, 3.82 [05:02:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 12.00, 10.59, 8.98 [05:04:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.65, 10.12, 8.99 [05:05:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.27, 1.64 [05:06:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.38, 4.24, 3.98 [05:08:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.92, 10.04, 9.98 [05:09:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.79, 1.84 [05:11:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.79, 1.83 [05:15:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.48, 3.99 [05:16:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.62, 3.97 [05:17:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.85, 2.24, 2.00 [05:18:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.89, 4.03 [05:20:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.63, 3.91 [05:20:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.97, 10.19, 9.74 [05:23:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.52, 1.44, 1.26 [05:24:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.45, 3.90 [05:24:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 10.75, 10.09 [05:25:07] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.77, 1.42 [05:25:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.11, 4.44, 4.06 [05:26:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.99, 4.06 [05:27:06] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.55, 1.70, 1.44 [05:28:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 3.76, 3.73 [05:28:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.46, 10.09, 9.97 [05:31:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 2.07, 1.65 [05:33:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.69, 1.57 [05:33:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.58, 2.18, 1.60 [05:34:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.63, 11.84, 10.65 [05:35:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.27, 1.84, 1.55 [05:36:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.44, 11.91, 10.84 [05:39:03] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.60, 1.50, 1.50 [05:46:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.37, 9.65, 10.12 [05:52:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.91, 1.51, 1.34 [05:54:08] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=iceriawiki --no-updates /mnt/mediawiki-static/metawiki/ImportDump/iceriawiki-20220724205902.xml (START) [05:54:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:54:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.30, 1.28 [05:55:06] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=iceriawiki --no-updates /mnt/mediawiki-static/metawiki/ImportDump/iceriawiki-20220724205902.xml (END - exit=0) [05:55:07] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=iceriawiki (START) [05:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.88, 11.94, 10.78 [06:04:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.46, 10.52, 9.42 [06:06:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.31, 9.85, 9.30 [06:06:47] [02mw-config] 07AgentIsai opened pull request 03#4828: Revert special web badge - 13https://github.com/miraheze/mw-config/pull/4828 [06:06:47] [url] Page not found · GitHub · GitHub | github.com [06:07:23] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:31] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:36] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=iceriawiki (END - exit=0) [06:07:36] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:37] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=iceriawiki --active --update (END - exit=0) [06:07:53] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:58] miraheze/mw-config - AgentIsai the build passed. [06:08:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:08:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:09:17] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.328 second response time [06:09:30] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.324 second response time [06:09:33] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28496 bytes in 0.832 second response time [06:09:50] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.026 second response time [06:12:46] [discord] @Reception123 around to merge? ^ [06:14:38] sure [06:14:50] [02mw-config] 07Reception123 closed pull request 03#4828: Revert special web badge - 13https://github.com/miraheze/mw-config/pull/4828 [06:14:50] [url] Page not found · GitHub · GitHub | github.com [06:14:51] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d165f7b9f4b7...3616991dcd89 [06:14:52] [url] Comparing d165f7b9f4b7...3616991dcd89 · miraheze/mw-config · GitHub | github.com [06:14:53] [02miraheze/mw-config] 07AgentIsai 033616991 - Revert special web badge (#4828) [06:15:05] !log [reception@mwtask111] starting deploy of {'config': True} to all [06:15:12] [discord] Thanks [06:15:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:29] !log [reception@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 23s [06:15:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:58] miraheze/mw-config - Reception123 the build passed. [06:17:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 10.23, 9.46 [06:18:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.50, 10.39, 9.79 [06:18:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.57, 11.03, 9.84 [06:19:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.21, 9.63, 8.87 [06:20:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.59, 11.15, 9.39 [06:20:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.79, 10.35, 9.75 [06:21:27] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:53] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.97, 8.85, 8.67 [06:22:07] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:10] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 10.39, 9.33 [06:22:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.24, 10.20, 9.91 [06:22:40] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:51] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:22:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.13, 9.97, 9.68 [06:23:02] !log [@mwtask111] starting deploy of {'config': True} to all [06:23:30] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 27s [06:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:24:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.07, 10.05, 9.33 [06:25:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.38, 1.72, 1.45 [06:26:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [06:26:06] !log [@test101] starting deploy of {'config': True} to all [06:26:07] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [06:26:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.47, 1.39 [06:28:29] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.813 second response time [06:28:31] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.279 second response time [06:30:17] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.885 second response time [06:31:07] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.172 second response time [06:31:41] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.988 second response time [06:32:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.96, 10.46, 10.00 [06:34:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.44, 11.39, 10.42 [06:36:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 10.48, 10.21 [06:38:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.03, 10.09, 10.10 [06:46:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.25, 10.36, 9.63 [06:48:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.42, 10.16, 9.98 [06:50:50] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:52:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.49, 9.89, 9.93 [06:52:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.22, 9.87, 9.68 [06:58:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.78, 9.66, 8.96 [06:59:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.18, 10.37, 10.07 [07:00:50] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw111 [07:00:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.14, 9.72, 9.06 [07:03:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.33, 10.88, 10.31 [07:05:33] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw101 mw112 [07:06:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.29, 10.91, 11.84 [07:07:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 1.45, 1.26 [07:08:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.96, 10.45, 9.32 [07:08:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.47, 11.65, 11.98 [07:09:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.32, 1.35, 1.25 [07:10:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.12, 10.76, 9.55 [07:10:45] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [07:13:33] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:14:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.42, 10.79, 9.91 [07:14:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.85, 9.78, 9.50 [07:14:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.75, 11.99, 11.51 [07:15:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [07:15:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.33, 10.24, 9.61 [07:16:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.49, 3.32, 3.94 [07:18:17] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.86, 9.69, 9.68 [07:18:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.69, 10.08, 9.71 [07:18:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.44, 1.17 [07:18:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.76, 1.48 [07:20:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.55, 4.06, 4.06 [07:20:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.50, 1.67, 1.48 [07:22:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.02, 10.45, 9.98 [07:22:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.26, 11.39, 10.28 [07:22:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.23, 1.36, 1.20 [07:23:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.46, 11.40, 10.37 [07:24:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.26, 9.75, 9.79 [07:24:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.71, 3.93 [07:24:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.54, 3.93 [07:24:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.39, 10.22, 9.99 [07:24:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:25:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.42, 11.00, 10.35 [07:26:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [07:27:48] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 4.04, 3.99 [07:28:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.89, 3.97 [07:28:34] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.50, 9.60, 9.82 [07:30:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.55, 3.82 [07:30:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.91, 11.19, 10.87 [07:31:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.71, 9.49, 9.96 [07:32:06] PROBLEM - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 4.00, 3.95 [07:32:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.18, 10.70, 10.19 [07:32:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.53, 10.96, 10.83 [07:33:45] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [07:34:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.31, 10.89, 10.32 [07:34:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.27, 11.61, 11.09 [07:34:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.15, 10.54, 11.95 [07:35:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.87, 10.23, 10.14 [07:36:06] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [07:36:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.94, 3.93 [07:36:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.44, 10.19, 10.13 [07:36:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.07, 11.40, 11.08 [07:37:20] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.426 second response time [07:38:06] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:38:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.91, 3.91 [07:39:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.53, 11.22, 10.56 [07:39:37] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [07:41:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.78, 11.35, 10.68 [07:42:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.89, 10.72, 10.03 [07:42:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.72, 10.30, 10.10 [07:43:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.70, 11.53, 10.81 [07:44:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.87, 3.92 [07:45:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.74, 10.36, 10.49 [07:46:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.22, 9.81, 9.87 [07:46:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.22, 4.49, 4.15 [07:46:27] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.32, 9.49, 9.85 [07:48:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.12, 3.69, 3.90 [07:48:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.20, 8.30, 10.10 [07:49:22] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw111 mw122 [07:49:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.94, 3.04, 3.79 [07:50:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.97, 4.02, 4.01 [07:50:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.01, 3.15, 3.78 [07:51:10] PROBLEM - wiki.timewisely.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.70, 9.85, 8.61 [07:52:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.09, 3.33, 3.76 [07:53:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.09, 11.37, 10.78 [07:54:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.54, 8.58, 10.00 [07:55:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.72, 10.42, 10.50 [07:55:47] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [07:55:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.22, 2.66, 3.38 [07:57:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.02, 10.07, 9.15 [07:57:46] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:58:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 2.78, 3.36 [07:58:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.66, 11.00, 10.62 [07:59:40] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:00:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.00, 10.21, 10.38 [08:00:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.25, 11.11, 10.21 [08:01:05] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.53, 10.95, 10.61 [08:01:43] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw101 mw112 mw122 [08:01:49] RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Sat 01 Oct 2022 16:03:14 GMT +0000. [08:02:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.94, 11.17, 10.39 [08:02:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.42, 3.50 [08:03:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.35, 10.76, 10.62 [08:03:53] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [08:04:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.20, 10.59, 9.90 [08:04:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.74, 9.61, 9.92 [08:04:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.54, 3.53 [08:05:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.96, 9.54, 10.20 [08:05:26] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:39] PROBLEM - smashloop.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.60, 9.99, 9.76 [08:06:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.46, 3.49 [08:06:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.79, 11.37, 10.81 [08:07:23] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.027 second response time [08:08:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.78, 10.70, 10.23 [08:08:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.82, 3.55, 3.56 [08:08:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.84, 10.83, 10.75 [08:10:00] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [08:10:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.29, 11.04, 10.29 [08:10:09] PROBLEM - sims.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.81, 11.33, 10.55 [08:10:24] PROBLEM - wiki.rtapp.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 10.84, 10.78 [08:12:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.12, 11.53, 10.57 [08:12:04] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:12:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:12:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.81, 3.62 [08:12:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.65, 9.89, 10.11 [08:12:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.04, 11.97, 11.21 [08:13:54] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw121 [08:14:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.76, 10.30, 10.25 [08:14:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.47, 3.52 [08:14:17] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.034 second response time [08:14:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.54, 3.52 [08:14:39] PROBLEM - hellointernet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.44, 8.52, 9.60 [08:16:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.45, 3.49 [08:16:37] PROBLEM - mwcosmos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.88, 4.31, 3.77 [08:18:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.54, 3.55 [08:19:54] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.49, 2.99, 3.30 [08:20:23] RECOVERY - wiki.timewisely.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.timewisely.net' will expire on Mon 22 Aug 2022 22:01:18 GMT +0000. [08:20:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.42, 3.45 [08:21:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [08:22:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 2.97, 3.29 [08:23:49] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [08:23:50] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:24:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 11.44, 11.70 [08:25:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.89, 10.19, 9.70 [08:26:06] PROBLEM - hi.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:27:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.21, 11.13, 10.11 [08:27:38] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.32, 3.41 [08:29:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [08:29:48] PROBLEM - squareenix.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:52] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw111 mw112 mw122 [08:30:28] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.67, 3.53 [08:30:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.87, 8.54, 9.75 [08:31:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.36, 11.76, 10.66 [08:31:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.47, 11.90, 10.72 [08:32:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.32, 7.90, 9.95 [08:33:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.18, 11.97, 10.87 [08:34:22] RECOVERY - smashloop.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'smashloop.wiki' will expire on Wed 31 Aug 2022 22:13:31 GMT +0000. [08:34:25] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:34:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.69, 3.59 [08:35:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.12, 11.13, 10.56 [08:35:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.88, 10.39, 10.43 [08:36:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.98, 9.77, 9.23 [08:36:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 3.13, 3.39 [08:36:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.52, 2.75, 3.30 [08:36:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.48, 11.08, 10.44 [08:37:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.46, 9.45, 10.08 [08:38:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.88, 9.70, 10.03 [08:38:54] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:39:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.95, 9.52, 10.16 [08:39:15] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:39:25] RECOVERY - wiki.rtapp.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rtapp.tk' will expire on Sun 14 Aug 2022 00:07:41 GMT +0000. [08:40:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.19, 10.89, 9.82 [08:40:38] PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:10] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.563 second response time [08:41:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.95, 8.97, 8.44 [08:42:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.67, 8.86, 9.20 [08:43:24] RECOVERY - hellointernet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:43:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.28, 10.57, 9.09 [08:44:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.70, 9.84, 9.75 [08:45:38] RECOVERY - mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Aug 2022 15:03:04 GMT +0000. [08:45:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.83, 10.21, 9.18 [08:45:57] PROBLEM - hoodwiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:57] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.33, 10.67, 10.06 [08:47:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [08:47:34] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [08:47:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.66, 9.35, 8.99 [08:48:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.33, 9.92, 9.85 [08:49:14] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:49:20] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:51:11] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.90, 0.97, 0.86 [08:52:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.72, 3.40 [08:52:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.73, 3.33, 3.06 [08:53:33] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [08:54:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.89, 3.67, 3.42 [08:55:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.69, 10.50, 9.64 [08:55:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.51, 11.06, 9.91 [08:55:45] RECOVERY - hi.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.famepedia.org' will expire on Tue 18 Oct 2022 14:04:28 GMT +0000. [08:55:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 1.90, 1.36 [08:56:25] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.84, 3.32 [08:57:27] RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Tue 23 Aug 2022 09:06:44 GMT +0000. [08:57:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 11.32, 10.04 [08:57:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.46, 10.94, 10.02 [08:58:03] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.65, 4.01, 3.60 [08:58:38] RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:59:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.85, 1.46 [09:00:04] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [09:01:15] PROBLEM - www.opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:34] PROBLEM - cp21 Varnish Backends on cp21 is WARNING: No backends detected. If this is an error, see readme.txt [09:01:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.32, 9.50, 9.66 [09:01:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.87, 1.54, 1.40 [09:02:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.21, 2.25, 1.51 [09:03:20] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.78, 2.69, 1.56 [09:03:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.66, 4.92, 3.86 [09:03:33] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 4 backends are down. mw101 mw102 mw112 mw121 [09:04:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.94, 11.16, 10.08 [09:04:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.35, 1.91, 1.47 [09:04:43] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.036 second response time [09:05:27] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:29] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.93, 1.97, 1.44 [09:05:38] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:39] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.64, 11.06, 10.18 [09:06:16] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.56, 1.39 [09:07:05] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:24] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.02, 2.39, 1.68 [09:07:25] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.014 second response time [09:07:42] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.900 second response time [09:07:42] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.948 second response time [09:07:53] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.080 second response time [09:08:41] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:15] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.248 second response time [09:09:22] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.19, 1.89, 1.58 [09:09:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.25, 11.22, 11.01 [09:09:40] RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Tue 23 Aug 2022 00:16:11 GMT +0000. [09:10:25] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [09:10:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:10:37] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:54] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:11:16] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.64, 1.49, 1.46 [09:11:56] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [09:11:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.39, 12.07, 10.77 [09:11:59] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:04] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 5 % full [09:13:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.87, 11.00, 9.41 [09:13:43] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:28] PROBLEM - wiki.worldbuilding.network - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:30] RECOVERY - hoodwiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'hoodwiki.xyz' will expire on Wed 19 Oct 2022 06:37:38 GMT +0000. [09:14:32] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 04 Oct 2022 18:56:56 GMT +0000. [09:15:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.26, 8.37, 9.91 [09:15:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.35, 11.55, 9.80 [09:17:30] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.765 second response time [09:18:25] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.642 second response time [09:19:23] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.135 second response time [09:19:36] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 11.49, 11.12 [09:20:42] PROBLEM - cp20 Stunnel HTTP for mon111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:56] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [09:21:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.08, 11.04, 10.26 [09:21:12] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:13] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:17] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.489 second response time [09:21:41] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.055 second response time [09:21:43] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.059 second response time [09:21:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.52, 10.08, 9.09 [09:22:10] PROBLEM - threedomwiki.pcast.site - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:25] PROBLEM - wiki.leapit.no - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:41] PROBLEM - vrcdev.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.23, 9.80, 9.90 [09:23:06] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 28 minutes ago with 0 failures [09:23:39] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:45] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.25, 8.54, 8.64 [09:24:50] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.35, 9.13, 8.07 [09:25:03] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 9.74, 9.45 [09:25:13] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:15] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:37] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.150 second response time [09:26:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.80, 9.57, 8.38 [09:27:03] RECOVERY - cp20 Stunnel HTTP for mon111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 37927 bytes in 1.062 second response time [09:27:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.82, 7.74, 9.55 [09:27:41] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:48] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.841 second response time [09:30:00] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:03] RECOVERY - www.opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Thu 01 Sep 2022 00:46:11 GMT +0000. [09:30:04] PROBLEM - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:06] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.82, 10.96, 10.26 [09:32:10] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.356 second response time [09:32:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.97, 11.13, 10.40 [09:33:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.64, 11.44, 10.55 [09:33:04] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.117 second response time [09:33:21] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.327 second response time [09:33:36] PROBLEM - pornwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:33:37] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:33:40] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.321 second response time [09:33:59] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.69, 11.68, 10.69 [09:34:52] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.524 second response time [09:35:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.21, 9.17, 9.82 [09:35:30] PROBLEM - arrieretoile.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:35] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.546 second response time [09:35:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.56, 9.12, 8.25 [09:35:54] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.90, 1.79, 1.44 [09:35:57] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:36:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.05, 10.28, 10.32 [09:36:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:37:54] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.38, 1.67, 1.44 [09:38:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.50, 8.83, 9.76 [09:38:27] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:58] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:40:52] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.141 second response time [09:42:02] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.564 second response time [09:43:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.02, 9.73, 9.06 [09:43:50] RECOVERY - wiki.worldbuilding.network - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldbuilding.network' will expire on Mon 10 Oct 2022 05:23:51 GMT +0000. [09:43:57] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.12, 1.78, 1.58 [09:44:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.66, 10.69, 9.90 [09:44:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.33, 10.38, 9.31 [09:45:48] PROBLEM - scpvncommunitywiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.43, 1.29, 1.42 [09:46:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.71, 9.69, 9.64 [09:46:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.67, 9.97, 9.32 [09:47:28] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.24, 11.05, 9.74 [09:48:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.28, 9.42, 9.18 [09:48:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.36, 10.46, 8.98 [09:49:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.18, 10.29, 9.64 [09:49:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.24, 10.55, 9.36 [09:49:51] PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.56, 10.24, 9.07 [09:50:36] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Sat 01 Oct 2022 16:09:21 GMT +0000. [09:51:21] PROBLEM - wiki.meeusen.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.48, 8.75, 9.14 [09:51:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.12, 10.06, 9.33 [09:52:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.80, 10.32, 9.66 [09:52:07] RECOVERY - vrcdev.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vrcdev.wiki' will expire on Tue 20 Sep 2022 12:25:22 GMT +0000. [09:54:00] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Mon 22 Aug 2022 22:44:29 GMT +0000. [09:54:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.17, 8.94, 9.24 [09:54:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [09:54:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.50, 11.19, 10.14 [09:56:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:56:10] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:56:28] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:57:24] PROBLEM - cp21 Stunnel HTTP for mon111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:08] PROBLEM - crocwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:22] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [10:00:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [10:00:11] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.24, 9.36, 9.39 [10:00:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.88, 10.50, 9.39 [10:01:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.28, 1.53, 1.99 [10:03:32] RECOVERY - pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pornwiki.org' will expire on Sat 17 Sep 2022 18:32:03 GMT +0000. [10:03:51] PROBLEM - wiki.auraxis.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:03] RECOVERY - arrieretoile.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'arrieretoile.fr' will expire on Thu 13 Oct 2022 04:37:30 GMT +0000. [10:04:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:04:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.97, 9.78, 9.38 [10:05:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.54, 11.01, 9.97 [10:05:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.87, 2.00 [10:06:31] RECOVERY - cp21 Stunnel HTTP for mon111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 37919 bytes in 0.032 second response time [10:06:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.82, 11.19, 11.01 [10:07:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.25, 9.11, 9.39 [10:07:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.82, 1.97 [10:09:05] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.29, 10.14, 9.07 [10:09:38] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:47] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.36, 8.45, 9.96 [10:11:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.40, 10.01, 9.18 [10:11:34] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:48] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.034 second response time [10:11:53] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 5.144 second response time [10:11:53] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 4.373 second response time [10:12:31] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [10:12:35] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [10:13:30] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.318 second response time [10:13:38] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:13:50] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [10:14:20] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [10:14:27] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.252 second response time [10:14:33] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw101 [10:14:35] PROBLEM - cp20 Stunnel HTTP for mon111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:14:45] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [10:14:48] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.38, 3.95 [10:16:33] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [10:16:35] RECOVERY - cp20 Stunnel HTTP for mon111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 37912 bytes in 0.041 second response time [10:16:43] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.533 second response time [10:16:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.29, 10.23, 9.20 [10:17:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.06, 2.02, 1.93 [10:18:12] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [10:18:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.74, 8.81, 8.25 [10:18:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.40, 9.31, 9.00 [10:19:18] RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Fri 30 Sep 2022 17:38:06 GMT +0000. [10:19:51] RECOVERY - threedomwiki.pcast.site - LetsEncrypt on sslhost is OK: OK - Certificate 'threedomwiki.pcast.site' will expire on Mon 22 Aug 2022 22:02:33 GMT +0000. [10:20:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.62, 8.39, 8.17 [10:20:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.07, 9.33, 8.59 [10:20:31] PROBLEM - en.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:32] RECOVERY - wiki.leapit.no - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.leapit.no' will expire on Wed 17 Aug 2022 08:40:55 GMT +0000. [10:20:46] RECOVERY - wiki.meeusen.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Sat 17 Sep 2022 21:26:27 GMT +0000. [10:21:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 12.00, 9.60, 8.98 [10:21:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.53, 3.90 [10:22:02] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw102 mw112 mw122 [10:22:27] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.17, 8.46, 8.39 [10:23:29] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [10:23:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.55, 10.47, 9.38 [10:24:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.48, 3.91 [10:25:16] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:25:53] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:12] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.031 second response time [10:27:31] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:27:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.54, 10.24, 9.63 [10:27:49] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.324 second response time [10:27:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.64, 3.77 [10:27:59] RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Thu 15 Sep 2022 05:33:41 GMT +0000. [10:28:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.71, 3.95, 3.82 [10:28:24] RECOVERY - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipedia.yogfront.ooo' will expire on Mon 22 Aug 2022 21:48:31 GMT +0000. [10:28:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 3.66, 3.87 [10:29:47] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.08, 9.00, 9.24 [10:29:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.79, 3.82 [10:30:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:30:21] PROBLEM - wiki-asterix.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.82, 4.64, 4.12 [10:32:30] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:46] RECOVERY - wiki.auraxis.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.auraxis.co' will expire on Fri 21 Oct 2022 06:34:16 GMT +0000. [10:34:57] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:17] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:20] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.74, 3.89 [10:36:54] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.832 second response time [10:37:14] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [10:37:18] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.037 second response time [10:37:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.29, 10.42, 9.85 [10:37:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 4.12, 4.02 [10:39:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.61, 9.31, 9.51 [10:40:09] PROBLEM - sims.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:09] PROBLEM - test2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:15] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:41:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.63, 1.14 [10:42:20] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.581 second response time [10:43:09] PROBLEM - www.mcpk.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.42, 10.70, 10.06 [10:43:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.46, 1.55, 1.17 [10:44:39] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:44:48] PROBLEM - experimyco.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:48] RECOVERY - scpvncommunitywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'scpvncommunitywiki.xyz' will expire on Tue 23 Aug 2022 02:34:15 GMT +0000. [10:46:08] PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.70, 1.32 [10:48:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:49:05] RECOVERY - en.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Sat 17 Sep 2022 22:00:09 GMT +0000. [10:49:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.48, 1.28 [10:49:54] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:51:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.26, 8.94, 9.80 [10:51:41] PROBLEM - wikiru.wildterra2.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [10:52:25] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:00] PROBLEM - cnt.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:28] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.867 second response time [10:55:01] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [10:55:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.55, 9.84, 8.53 [10:55:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.79, 10.97, 8.84 [10:56:12] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [10:56:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.37, 9.61, 8.60 [10:57:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.79, 9.66, 8.61 [10:58:06] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [10:58:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:58:17] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:59:05] PROBLEM - slupe.fp64.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:08] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw112 mw121 [10:59:15] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:59:40] PROBLEM - scum-wiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:48] RECOVERY - wiki-asterix.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki-asterix.cf' will expire on Sat 17 Sep 2022 20:36:30 GMT +0000. [10:59:48] PROBLEM - en.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:16] PROBLEM - cp20 Stunnel HTTP for mon111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.84, 9.59, 8.62 [11:00:43] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.027 second response time [11:01:03] RECOVERY - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Fri 16 Sep 2022 12:56:22 GMT +0000. [11:01:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.88, 10.01, 9.38 [11:01:40] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.58, 0.69, 0.64 [11:01:56] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw111 mw121 mw122 [11:02:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.43, 9.65, 8.77 [11:02:42] PROBLEM - celeste.ink - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:46] RECOVERY - cp20 Stunnel HTTP for mon111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 37911 bytes in 0.044 second response time [11:03:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.16, 9.47, 9.28 [11:03:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.42, 3.95 [11:04:58] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:59] PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:06:33] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw102 [11:06:53] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.051 second response time [11:07:37] PROBLEM - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.47, 3.92 [11:08:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.49, 3.87 [11:08:33] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [11:08:54] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:08:54] RECOVERY - test2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:09:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 9.96, 8.83 [11:09:31] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:01] PROBLEM - it.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:12] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:11:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.10, 9.72, 8.89 [11:11:31] [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/miraheze/ErrorPages/compare/39acc1161114...c5ee580377f0 [11:11:31] [url] Comparing 39acc1161114...c5ee580377f0 · miraheze/ErrorPages · GitHub | github.com [11:11:32] [02miraheze/ErrorPages] 07translatewiki 03c5ee580 - Localisation updates from https://translatewiki.net. [11:11:32] [url] Main page - translatewiki.net | translatewiki.net [11:11:34] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/9896519dc8b7...cd717d5bff71 [11:11:34] [url] Comparing 9896519dc8b7...cd717d5bff71 · miraheze/landing · GitHub | github.com [11:11:35] [02miraheze/landing] 07translatewiki 03cd717d5 - Localisation updates from https://translatewiki.net. [11:11:35] [url] Main page - translatewiki.net | translatewiki.net [11:11:37] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/ab596c6fdf87...3c82cd84b8e6 [11:11:37] [url] Comparing ab596c6fdf87...3c82cd84b8e6 · miraheze/CreateWiki · GitHub | github.com [11:11:38] [02miraheze/CreateWiki] 07translatewiki 033c82cd8 - Localisation updates from https://translatewiki.net. [11:11:38] [url] Main page - translatewiki.net | translatewiki.net [11:11:40] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/WikiDiscover/compare/d011591989c9...9a832a767850 [11:11:40] [url] Comparing d011591989c9...9a832a767850 · miraheze/WikiDiscover · GitHub | github.com [11:11:41] [02miraheze/WikiDiscover] 07translatewiki 039a832a7 - Localisation updates from https://translatewiki.net. [11:11:41] [url] Main page - translatewiki.net | translatewiki.net [11:11:43] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+3/-0/±3] 13https://github.com/miraheze/MirahezeMagic/compare/2a7be60118a1...fa2bf9b6c2f4 [11:11:43] [url] Comparing 2a7be60118a1...fa2bf9b6c2f4 · miraheze/MirahezeMagic · GitHub | github.com [11:11:44] [02miraheze/MirahezeMagic] 07translatewiki 03fa2bf9b - Localisation updates from https://translatewiki.net. [11:11:44] [url] Main page - translatewiki.net | translatewiki.net [11:12:01] RECOVERY - www.mcpk.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mcpk.wiki' will expire on Sat 17 Sep 2022 18:44:27 GMT +0000. [11:12:28] miraheze/ErrorPages - translatewiki the build passed. [11:13:04] miraheze/landing - translatewiki the build passed. [11:13:05] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [11:13:24] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [11:13:31] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:13:42] RECOVERY - experimyco.com - LetsEncrypt on sslhost is OK: OK - Certificate 'experimyco.com' will expire on Thu 15 Sep 2022 05:36:04 GMT +0000. [11:14:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.28, 10.01, 9.07 [11:14:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.54, 3.74 [11:14:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.52, 3.71 [11:15:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.15, 9.00, 8.37 [11:15:50] RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Sun 14 Aug 2022 15:58:21 GMT +0000. [11:15:56] PROBLEM - commonswiki.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:58] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:16:07] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.56, 9.46, 8.98 [11:16:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.57, 3.73 [11:16:43] PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:59] PROBLEM - en.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:26] PROBLEM - wiki.digitalefitheid.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:18:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.41, 3.45, 3.70 [11:18:51] miraheze/WikiDiscover - translatewiki the build passed. [11:19:20] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:19:21] PROBLEM - nl.xliving.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.74, 8.68, 8.47 [11:20:21] miraheze/CreateWiki - translatewiki the build has errored. [11:20:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.70, 3.75 [11:20:36] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:05] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [11:21:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.01, 10.53, 9.03 [11:21:32] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 2.88, 3.35 [11:22:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.43, 3.64 [11:22:31] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:22:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.96, 11.33, 9.93 [11:23:06] PROBLEM - kiseki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:06] RECOVERY - cnt.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:23:31] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.057 second response time [11:23:48] miraheze/MirahezeMagic - translatewiki the build passed. [11:23:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.04, 2.79, 3.34 [11:24:16] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:55] PROBLEM - wiki.yahyabd.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:13] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:13] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:17] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:19] PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:20] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 2.78, 3.32 [11:26:34] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:36] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[Landing Sync],Exec[ErrorPages Sync] [11:27:25] PROBLEM - replication-ops.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:38] RECOVERY - en.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:04:52 GMT +0000. [11:28:43] RECOVERY - slupe.fp64.org - LetsEncrypt on sslhost is OK: OK - Certificate 'slupe.fp64.org' will expire on Tue 04 Oct 2022 18:47:51 GMT +0000. [11:30:03] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.878 second response time [11:30:10] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [11:30:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.81, 3.54, 3.44 [11:30:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.59, 3.53 [11:30:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.37, 11.50, 10.71 [11:31:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.46, 3.43 [11:32:22] RECOVERY - celeste.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Sat 24 Sep 2022 13:42:28 GMT +0000. [11:32:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.61, 3.54 [11:32:44] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.488 second response time [11:32:49] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.843 second response time [11:32:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.47, 9.80, 10.16 [11:33:09] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:33:29] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.555 second response time [11:34:09] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw121 [11:34:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.92, 4.19, 3.68 [11:34:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 3.86, 3.63 [11:34:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.30, 10.43, 9.09 [11:35:06] RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Mon 22 Aug 2022 23:45:19 GMT +0000. [11:36:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.51, 3.49 [11:36:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.24, 10.01, 9.10 [11:37:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.24, 1.57, 1.93 [11:37:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.64, 4.57, 3.91 [11:38:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 4.08, 3.70 [11:38:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.67, 3.60 [11:38:45] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.856 second response time [11:40:19] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.069 second response time [11:40:51] RECOVERY - it.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'it.famepedia.org' will expire on Sat 08 Oct 2022 06:08:33 GMT +0000. [11:40:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.97, 10.58, 10.13 [11:42:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.30, 4.35, 3.86 [11:42:51] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.38, 0.96 [11:42:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.94, 9.97, 9.13 [11:42:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.48, 9.91, 9.96 [11:44:10] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [11:44:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.70, 9.61, 9.12 [11:44:48] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.79, 1.10, 0.91 [11:44:50] RECOVERY - commonswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:44:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.47, 11.99, 11.85 [11:46:01] RECOVERY - vitriol.spoon.army - LetsEncrypt on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Tue 23 Aug 2022 01:40:06 GMT +0000. [11:46:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.22, 3.71, 3.78 [11:46:45] RECOVERY - en.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Mon 10 Oct 2022 05:45:28 GMT +0000. [11:47:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.73, 1.81, 1.81 [11:47:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 4.00, 3.94 [11:48:00] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [11:48:16] RECOVERY - nl.xliving.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nl.xliving.org' will expire on Tue 06 Sep 2022 20:54:18 GMT +0000. [11:48:17] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [11:48:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.95, 12.08, 11.88 [11:49:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [11:49:08] PROBLEM - otcg.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.52, 4.42, 4.09 [11:50:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:50:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.45, 4.48, 4.07 [11:50:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.30, 10.82, 11.43 [11:51:21] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:46] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:01] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw101 mw102 mw111 [11:52:10] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.64, 3.81 [11:53:22] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Mon 10 Oct 2022 04:52:56 GMT +0000. [11:53:31] PROBLEM - trollpasta.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:43] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [11:53:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.43, 9.65, 8.51 [11:53:50] PROBLEM - ambient.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.59, 3.85 [11:54:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [11:54:10] RECOVERY - cp21 Stunnel HTTP for puppet111 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.027 second response time [11:54:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.67, 3.84 [11:54:39] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [11:55:18] RECOVERY - wiki.worldsofweary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldsofweary.com' will expire on Sat 17 Sep 2022 21:51:04 GMT +0000. [11:55:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.48, 9.08, 8.46 [11:56:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:56:47] PROBLEM - commons.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.75, 11.83, 11.58 [11:57:10] RECOVERY - replication-ops.com - LetsEncrypt on sslhost is OK: OK - Certificate 'replication-ops.com' will expire on Tue 18 Oct 2022 15:32:39 GMT +0000. [11:57:22] RECOVERY - scum-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'scum-wiki.com' will expire on Sat 22 Oct 2022 05:35:09 GMT +0000. [11:57:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.98, 2.00 [11:58:52] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.25, 10.70, 9.75 [11:59:25] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:29] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:33] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:01] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:00:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.73, 2.75, 3.32 [12:00:46] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:40] PROBLEM - istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:55] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [12:02:15] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.174 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [12:02:45] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.035 second response time [12:02:54] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.648 second response time [12:03:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.47, 10.07, 9.75 [12:03:32] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.959 second response time [12:03:38] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.879 second response time [12:04:06] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.074 second response time [12:04:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.28, 2.87, 3.34 [12:05:44] PROBLEM - wikicompliance.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:06:19] RECOVERY - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Sat 17 Sep 2022 18:56:39 GMT +0000. [12:06:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.21, 10.54, 9.61 [12:07:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.58, 3.53 [12:07:42] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw101 mw121 [12:07:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.69, 3.00, 3.40 [12:08:05] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [12:08:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.24, 10.18, 9.61 [12:10:14] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:10:44] PROBLEM - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.56, 3.79, 3.60 [12:11:42] PROBLEM - notes.mridulpm.in - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.19, 4.08, 3.75 [12:12:46] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.083 second response time [12:13:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.32, 3.45 [12:13:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.59, 3.61 [12:15:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 3.06, 3.35 [12:15:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.97, 1.91 [12:15:52] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw111 mw121 [12:15:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.63, 3.61 [12:16:07] PROBLEM - commonswiki.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [12:16:25] RECOVERY - wiki.digitalefitheid.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitalefitheid.nl' will expire on Wed 21 Sep 2022 08:49:04 GMT +0000. [12:16:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.07, 10.42, 11.68 [12:17:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [12:17:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.10, 9.94, 8.36 [12:17:42] !log DELETE FROM oldimage WHERE oi_archive_name=''; (scruffywiki) [12:17:43] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [12:17:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.47, 3.55 [12:18:21] PROBLEM - journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.96, 10.27, 9.63 [12:18:48] RECOVERY - otcg.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'otcg.ml' will expire on Sat 17 Sep 2022 22:12:58 GMT +0000. [12:19:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.97, 8.55, 8.05 [12:19:52] RECOVERY - wikiru.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiru.wildterra2.com' will expire on Wed 24 Aug 2022 02:47:08 GMT +0000. [12:20:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.07, 11.16, 10.03 [12:20:41] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Wed 24 Aug 2022 04:55:08 GMT +0000. [12:20:47] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Tue 23 Aug 2022 08:39:11 GMT +0000. [12:20:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.84, 10.16, 11.15 [12:21:37] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [12:21:52] RECOVERY - kiseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kiseki.wiki' will expire on Wed 24 Aug 2022 06:15:40 GMT +0000. [12:21:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.27, 4.35, 3.83 [12:22:08] RECOVERY - trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Mon 10 Oct 2022 06:29:05 GMT +0000. [12:22:31] RECOVERY - wiki.yahyabd.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yahyabd.xyz' will expire on Sun 14 Aug 2022 20:06:30 GMT +0000. [12:22:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.54, 10.50, 9.92 [12:23:10] RECOVERY - ambient.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ambient.wiki' will expire on Mon 10 Oct 2022 05:15:24 GMT +0000. [12:23:34] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.14, 1.75, 1.17 [12:24:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.70, 9.23, 9.51 [12:24:46] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:16] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 1.61, 1.17 [12:25:36] RECOVERY - commons.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:46:28 GMT +0000. [12:26:06] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.37, 11.45, 9.76 [12:26:41] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.93, 1.30, 1.11 [12:27:41] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 5.758 second response time [12:28:03] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.217 second response time [12:28:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.65, 9.96, 9.45 [12:28:40] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.362 second response time [12:28:49] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.380 second response time [12:29:39] PROBLEM - microworldwiki.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.45, 3.26 [12:29:54] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [12:30:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.00, 11.06, 10.11 [12:30:56] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.48, 3.36 [12:31:15] PROBLEM - www.opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:27] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:38] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:46] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 3.18, 3.19 [12:32:30] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.45, 10.52, 11.24 [12:32:57] PROBLEM - www.zetapedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.13, 3.23, 3.28 [12:33:42] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [12:33:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.34, 3.65, 3.90 [12:34:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:35:06] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.383 second response time [12:35:32] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [12:35:38] RECOVERY - wikicompliance.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikicompliance.com' will expire on Fri 07 Oct 2022 11:39:30 GMT +0000. [12:35:43] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.427 second response time [12:36:22] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [12:36:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.33, 11.12, 10.62 [12:36:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.61, 11.73, 11.51 [12:37:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.80, 3.44 [12:37:48] PROBLEM - pt.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:31] PROBLEM - www.puzzles.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.46, 9.48, 10.08 [12:39:18] RECOVERY - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pablojprieto.com.es' will expire on Tue 23 Aug 2022 09:39:26 GMT +0000. [12:39:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 4.07, 3.96 [12:40:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [12:40:31] RECOVERY - notes.mridulpm.in - LetsEncrypt on sslhost is OK: OK - Certificate 'notes.mridulpm.in' will expire on Tue 20 Sep 2022 12:00:20 GMT +0000. [12:40:46] PROBLEM - en.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.45, 11.75, 11.57 [12:41:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 4.23, 3.71 [12:42:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:43:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.64, 10.24, 9.25 [12:43:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.25, 10.33, 9.10 [12:43:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.89, 3.95 [12:44:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.04, 9.45, 9.07 [12:45:01] RECOVERY - commonswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:45:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.54, 9.50, 8.95 [12:45:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.54, 3.58 [12:45:34] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [12:46:30] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.34, 10.53, 10.26 [12:47:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 3.69, 3.61 [12:47:46] RECOVERY - journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Wed 31 Aug 2022 18:13:33 GMT +0000. [12:47:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.60, 4.28, 4.06 [12:48:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.91, 9.18, 9.80 [12:50:28] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [12:50:40] PROBLEM - agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:20] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [12:54:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.39, 9.82, 9.32 [12:54:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.83, 8.36, 10.02 [12:55:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.73, 3.77 [12:56:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.11, 9.64, 8.09 [12:56:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.28, 8.48, 8.89 [12:57:09] PROBLEM - wiki.sketch.town - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 4.02, 3.86 [12:57:26] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.57, 9.10, 8.11 [12:59:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.98, 3.87 [12:59:21] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.023 second response time [13:00:02] RECOVERY - www.opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Thu 01 Sep 2022 00:46:11 GMT +0000. [13:00:21] RECOVERY - istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [13:00:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.28, 10.69, 10.27 [13:01:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.32, 4.12, 3.93 [13:01:56] RECOVERY - www.zetapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zetapedia.com' will expire on Wed 17 Aug 2022 21:08:43 GMT +0000. [13:02:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.70, 8.87, 9.65 [13:04:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.04, 10.85, 10.08 [13:05:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.92, 3.92 [13:05:30] PROBLEM - arrieretoile.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.53, 3.96 [13:06:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.40, 10.66, 9.47 [13:07:11] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Thu 29 Sep 2022 08:59:53 GMT +0000. [13:08:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.15, 10.36, 9.50 [13:08:19] RECOVERY - www.puzzles.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.puzzles.wiki' will expire on Thu 20 Oct 2022 16:22:25 GMT +0000. [13:08:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.62, 9.37, 9.81 [13:09:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [13:09:47] RECOVERY - en.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiyri.org' will expire on Wed 28 Sep 2022 05:13:53 GMT +0000. [13:09:47] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:09:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.88, 10.48, 9.19 [13:10:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.15, 8.87, 9.07 [13:10:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.64, 11.25, 10.42 [13:11:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.82, 3.81, 3.82 [13:11:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.74, 10.86, 9.49 [13:11:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 4.01, 3.98 [13:12:16] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.031 second response time [13:12:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.10, 11.13, 10.48 [13:12:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.25, 3.62, 3.75 [13:13:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.63, 3.49, 3.79 [13:14:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.95, 10.08, 9.82 [13:15:34] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Mon 22 Aug 2022 22:32:53 GMT +0000. [13:15:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.37, 1.71, 1.98 [13:15:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.02, 3.86, 3.88 [13:16:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.11, 9.64, 10.04 [13:16:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.17, 4.24, 3.95 [13:17:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.46, 2.04, 2.07 [13:19:11] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [13:19:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.83, 9.84, 9.66 [13:21:14] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 1.59, 1.14 [13:23:13] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.33, 1.44, 1.13 [13:24:41] PROBLEM - cp20 Stunnel HTTP for mon111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:24:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.78, 3.96 [13:25:17] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw101 mw102 mw122 [13:26:14] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [13:26:40] RECOVERY - cp20 Stunnel HTTP for mon111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 37927 bytes in 0.055 second response time [13:26:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.80, 12.09, 10.94 [13:27:16] RECOVERY - microworldwiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'microworldwiki.ga' will expire on Mon 26 Sep 2022 05:41:37 GMT +0000. [13:27:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.30, 1.62, 1.93 [13:27:39] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.92, 10.31, 9.29 [13:28:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.25, 4.08, 4.00 [13:29:40] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.067 second response time [13:30:02] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:12:19 GMT +0000. [13:30:27] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.31, 9.68, 9.19 [13:30:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.69, 3.86 [13:30:40] PROBLEM - popira.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.62, 11.92, 11.16 [13:32:49] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [13:32:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.85, 12.49, 11.47 [13:33:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.88, 3.15, 3.87 [13:33:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.74, 1.85 [13:33:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.52, 10.39, 8.98 [13:34:03] RECOVERY - arrieretoile.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'arrieretoile.fr' will expire on Thu 13 Oct 2022 04:37:30 GMT +0000. [13:34:14] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.51, 3.30, 2.90 [13:34:46] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [13:34:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.30, 11.14, 11.12 [13:35:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.08, 9.28, 8.76 [13:35:53] PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:12] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.25, 3.27, 2.94 [13:36:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:36:22] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:37:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.75, 1.84 [13:38:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.44, 10.29, 9.04 [13:38:33] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [13:38:36] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw101 mw122 [13:38:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.36, 8.95, 10.18 [13:38:59] PROBLEM - pt.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:05] PROBLEM - wikicompliance.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.37, 9.70, 8.99 [13:40:14] PROBLEM - source.erag.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.71, 3.42, 3.91 [13:43:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.27, 9.43, 8.00 [13:45:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.52, 3.68 [13:45:27] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.70, 10.77, 9.47 [13:47:03] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [13:47:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.40, 3.62 [13:47:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.80, 9.59, 8.43 [13:47:23] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.021 second response time [13:47:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.97, 1.88 [13:48:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.39, 10.02, 9.36 [13:49:21] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:57] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:36] RECOVERY - agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [13:50:54] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [13:51:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.26, 4.00, 3.84 [13:52:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [13:52:10] PROBLEM - threedomwiki.pcast.site - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:51] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:58] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [13:52:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.67, 3.63 [13:53:28] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:54:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.11, 4.52, 3.88 [13:54:55] PROBLEM - wiki.yahyabd.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:16] PROBLEM - wiki.jeremyboyd.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:59] RECOVERY - wiki.sketch.town - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sketch.town' will expire on Thu 13 Oct 2022 04:34:16 GMT +0000. [13:56:35] PROBLEM - rothwell-leeds.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:52] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw111 mw122 [13:57:16] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.56, 3.67 [13:59:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.36, 10.91, 9.58 [13:59:15] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.026 second response time [14:00:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.58, 9.54, 8.48 [14:00:05] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.054 second response time [14:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.74, 3.77 [14:00:21] PROBLEM - wiki-asterix.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:13] PROBLEM - wiki.slimeapedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.09, 3.74, 3.99 [14:01:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:02:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.66, 9.49, 8.59 [14:02:51] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [14:04:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.47, 10.83, 10.13 [14:05:01] RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Mon 22 Aug 2022 23:45:19 GMT +0000. [14:05:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.70, 3.89 [14:06:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.08, 10.15, 8.47 [14:06:56] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw102 [14:06:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.24, 9.34, 9.68 [14:07:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.64, 3.85 [14:08:20] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Thu 29 Sep 2022 08:59:53 GMT +0000. [14:08:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 10.03, 8.62 [14:09:44] RECOVERY - source.erag.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'source.erag.eu.org' will expire on Sun 09 Oct 2022 13:40:55 GMT +0000. [14:10:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.15, 3.39 [14:10:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.54, 9.43, 8.57 [14:11:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.89, 3.84, 3.85 [14:12:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.76, 1.29 [14:14:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.80, 1.36 [14:16:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 4.23, 3.80 [14:16:32] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 4.24, 3.79 [14:16:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.27, 2.06, 1.43 [14:18:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.91, 9.80, 8.85 [14:18:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.42, 1.80, 1.40 [14:18:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.54, 1.68, 1.42 [14:18:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.39, 10.92, 9.53 [14:19:30] PROBLEM - tensegritywiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:34] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Wed 24 Aug 2022 01:59:18 GMT +0000. [14:19:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.60, 10.09, 9.73 [14:19:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.98, 3.98 [14:20:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.85, 3.74 [14:20:47] PROBLEM - wiki.insideearth.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.63, 10.79, 9.65 [14:21:22] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:21:25] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.168 second response time [14:22:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.89, 3.43, 3.67 [14:22:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.89, 9.95, 9.17 [14:22:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.61, 1.79, 1.47 [14:23:23] PROBLEM - wiki.dereric.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:25] RECOVERY - wiki.yahyabd.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yahyabd.xyz' will expire on Sun 14 Aug 2022 20:06:30 GMT +0000. [14:24:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.59, 3.62 [14:24:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.71, 1.48 [14:25:05] RECOVERY - wiki.jeremyboyd.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jeremyboyd.org' will expire on Fri 12 Aug 2022 07:04:44 GMT +0000. [14:25:16] RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Sat 17 Sep 2022 18:56:39 GMT +0000. [14:25:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 4.08, 3.99 [14:26:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 3.97, 3.81 [14:26:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.49, 1.29, 1.35 [14:26:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.84, 11.62, 10.39 [14:26:56] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:28:12] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:28:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.92, 3.81 [14:28:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.55, 9.44, 9.75 [14:29:05] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.54, 1.00, 0.86 [14:29:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.55, 11.76, 10.65 [14:29:45] RECOVERY - wiki-asterix.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki-asterix.cf' will expire on Sat 17 Sep 2022 20:36:30 GMT +0000. [14:29:59] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [14:30:04] RECOVERY - wiki.slimeapedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimeapedia.org' will expire on Mon 10 Oct 2022 05:19:54 GMT +0000. [14:30:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.75, 3.77 [14:30:27] PROBLEM - cp20 Stunnel HTTP for mon111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.48, 10.27, 8.53 [14:30:43] RECOVERY - popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'popira.info' will expire on Sun 02 Oct 2022 11:12:21 GMT +0000. [14:31:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [14:31:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.23, 3.80, 3.98 [14:32:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.45, 9.62, 8.52 [14:32:43] RECOVERY - cp20 Stunnel HTTP for mon111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 37927 bytes in 2.078 second response time [14:32:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:33:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.13, 11.00, 10.59 [14:35:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 3.93, 3.96 [14:36:04] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [14:36:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.80, 3.74 [14:36:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.92, 9.25, 8.46 [14:37:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.99, 9.60, 10.09 [14:37:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.70, 3.87 [14:38:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.79, 3.74 [14:39:05] RECOVERY - wikicompliance.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikicompliance.com' will expire on Fri 07 Oct 2022 11:39:30 GMT +0000. [14:39:07] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.54, 9.45, 7.90 [14:40:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 3.55, 3.54 [14:40:28] PROBLEM - m33ks.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.76, 3.71 [14:40:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.06, 9.19, 8.68 [14:41:26] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:41:41] PROBLEM - avas.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.59, 8.88, 7.90 [14:42:20] [3422472.640766] Memory cgroup out of memory: Killed process 4031724 (convert) total-vm:267672kB, anon-rss:210736kB, file-rss:7636kB, shmem-rss:0kB, UID:33 pgtables:504kB oom_score_adj:0 [14:42:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.66, 3.69 [14:42:37] ^ paladox hmm seeing a lot of that on mw111 [14:42:46] I don't remember seeing cgroup ones before, just php childs being killed [14:42:59] could this explain all those spammy mw111 hund emails? [14:44:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.72, 3.66 [14:44:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.41, 9.97, 9.13 [14:45:09] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:51] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 5.663 second response time [14:46:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.83, 3.70 [14:46:27] PROBLEM - mwcosmos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.86, 3.76 [14:46:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.16, 9.99, 9.24 [14:47:10] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.053 second response time [14:47:27] PROBLEM - wiki-tintin.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.96, 3.78 [14:48:37] RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Tue 23 Aug 2022 08:10:17 GMT +0000. [14:49:06] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:49:21] PROBLEM - nl.xliving.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:51] RECOVERY - threedomwiki.pcast.site - LetsEncrypt on sslhost is OK: OK - Certificate 'threedomwiki.pcast.site' will expire on Mon 22 Aug 2022 22:02:33 GMT +0000. [14:49:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 3.84, 3.78 [14:50:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.98, 3.87 [14:50:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.54, 10.94, 9.48 [14:50:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.69, 11.45, 9.95 [14:50:43] RECOVERY - wiki.insideearth.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Tue 23 Aug 2022 08:05:34 GMT +0000. [14:51:27] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.758 second response time [14:52:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.39, 3.91, 3.74 [14:52:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.75, 10.92, 9.69 [14:52:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.49, 11.49, 10.17 [14:53:57] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.34, 9.22, 9.21 [14:54:36] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [14:54:59] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.51, 3.66 [14:56:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.93, 3.81 [14:56:16] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [14:56:26] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.065 second response time [14:56:32] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:57:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.54, 3.95, 3.81 [14:58:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.68, 4.09, 3.87 [14:58:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.53, 10.18, 10.00 [14:59:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.42, 1.70, 1.99 [14:59:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.83, 3.78 [15:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.89, 3.82 [15:00:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 10.20, 9.59 [15:00:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.67, 3.67 [15:02:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 4.03, 3.87 [15:02:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.61, 9.63, 9.46 [15:02:24] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw121 [15:02:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.32, 10.55, 10.22 [15:04:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.87, 3.83 [15:04:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.21, 3.50 [15:04:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.58, 11.49, 10.59 [15:06:07] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [15:06:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.03, 10.72, 10.41 [15:08:48] PROBLEM - az.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:01] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:09:26] RECOVERY - m33ks.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'm33ks.ml' will expire on Thu 29 Sep 2022 04:18:17 GMT +0000. [15:09:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 2.94, 3.37 [15:10:09] PROBLEM - test2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.30, 9.68, 10.06 [15:11:02] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.36, 0.40, 0.64 [15:11:28] RECOVERY - avas.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avas.wiki' will expire on Mon 10 Oct 2022 04:52:24 GMT +0000. [15:11:38] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.50, 1.43, 1.67 [15:12:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:12:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 2.73, 3.28 [15:12:24] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [15:13:09] PROBLEM - www.mcpk.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:54] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw121 mw122 [15:14:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.37, 1.94, 1.44 [15:14:39] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:16:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.19, 3.07, 3.31 [15:16:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.17, 1.56, 1.35 [15:16:47] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.252 second response time [15:17:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.43, 3.41 [15:18:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:18:13] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [15:18:17] RECOVERY - nl.xliving.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nl.xliving.org' will expire on Tue 06 Sep 2022 20:54:18 GMT +0000. [15:19:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.13, 2.14, 1.84 [15:19:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.07, 3.27 [15:21:49] PROBLEM - lab.wikimimo.com - reverse DNS on sslhost is WARNING: NXDOMAIN: The DNS query name does not exist: lab.wikimimo.com. [15:22:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.87, 9.72, 9.29 [15:22:45] PROBLEM - zenith.nsmbu.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.21, 3.27 [15:24:37] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:44:45 GMT +0000. [15:24:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.36, 8.95, 9.07 [15:25:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.01, 3.20, 3.26 [15:27:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 2.00, 1.91 [15:27:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 10.24, 9.56 [15:28:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.91, 9.37, 8.00 [15:28:38] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:29:41] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [15:29:42] PROBLEM - wiki.mcpirevival.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.42, 11.38, 10.06 [15:32:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.53, 10.12, 8.64 [15:33:29] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.048 second response time [15:33:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.59, 3.32 [15:33:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.25, 10.53, 10.05 [15:34:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.88, 3.55 [15:34:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.93, 3.55 [15:35:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.94, 3.49 [15:35:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.91, 10.04, 9.94 [15:36:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.38, 3.40 [15:37:19] RECOVERY - privacywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'privacywiki.xyz' will expire on Mon 26 Sep 2022 13:36:29 GMT +0000. [15:37:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 1.86, 1.84 [15:37:40] RECOVERY - az.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'az.wikiretet.ga' will expire on Tue 11 Oct 2022 04:40:37 GMT +0000. [15:37:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.73, 3.46 [15:38:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.83, 3.62 [15:38:54] RECOVERY - test2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:38:57] PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.82, 1.83 [15:40:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 4.30, 3.83 [15:41:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.91, 1.85 [15:41:55] [02mediawiki] 07Universal-Omega unlabeled pull request 03#8253: Bump skins/Cosmos from `48e1b8a` to `36df7a3` - 13https://github.com/miraheze/mediawiki/pull/8253 [15:41:55] [url] Page not found · GitHub · GitHub | github.com [15:42:06] [02mediawiki] 07Universal-Omega commented on pull request 03#8307: Bump skins/Cosmos from `48e1b8a` to `00f0f1e` - 13https://github.com/miraheze/mediawiki/pull/8307#issuecomment-1194237195 [15:42:06] [url] Bump skins/Cosmos from `48e1b8a` to `00f0f1e` by dependabot[bot] · Pull Request #8307 · miraheze/mediawiki · GitHub | github.com [15:42:09] [02mediawiki] 07dependabot[bot] edited pull request 03#8307: Bump skins/Cosmos from `48e1b8a` to `00f0f1e` - 13https://github.com/miraheze/mediawiki/pull/8307 [15:42:09] [url] Page not found · GitHub · GitHub | github.com [15:43:03] PROBLEM - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:09] [02mediawiki] 07dependabot[bot] edited pull request 03#8307: Bump skins/Cosmos from `48e1b8a` to `00f0f1e` - 13https://github.com/miraheze/mediawiki/pull/8307 [15:43:10] [url] Page not found · GitHub · GitHub | github.com [15:43:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [15:43:16] [url] Page not found · GitHub · GitHub | github.com [15:43:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Cosmos-c58ac13 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fcde2cae2a4f [15:43:19] [02miraheze/mediawiki] 07dependabot[bot] 03fcde2ca - Bump skins/Cosmos from `48e1b8a` to `c58ac13` [15:43:21] [02mediawiki] 07dependabot[bot] commented on pull request 03#8307: Bump skins/Cosmos from `48e1b8a` to `00f0f1e` - 13https://github.com/miraheze/mediawiki/pull/8307#issuecomment-1194240187 [15:43:21] [url] Bump skins/Cosmos from `48e1b8a` to `00f0f1e` by dependabot[bot] · Pull Request #8307 · miraheze/mediawiki · GitHub | github.com [15:43:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Cosmos-c58ac13 - 13https://github.com/miraheze/mediawiki [15:43:23] [url] Page not found · GitHub · GitHub | github.com [15:43:24] [02mediawiki] 07dependabot[bot] closed pull request 03#8307: Bump skins/Cosmos from `48e1b8a` to `00f0f1e` - 13https://github.com/miraheze/mediawiki/pull/8307 [15:43:24] ... [15:43:25] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-00f0f1e [15:43:27] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-00f0f1e - 13https://github.com/miraheze/mediawiki [15:43:27] ... [15:43:28] [02mediawiki] 07dependabot[bot] assigned pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [15:43:29] [url] Page not found · GitHub · GitHub | github.com [15:43:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.18, 3.19, 3.35 [15:43:40] PROBLEM - avas.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [15:43:48] ... [15:43:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [15:43:49] [url] Page not found · GitHub · GitHub | github.com [15:43:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [15:43:51] [url] Page not found · GitHub · GitHub | github.com [15:44:40] RECOVERY - mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Aug 2022 15:03:04 GMT +0000. [15:45:57] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:57] PROBLEM - hoodwiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.63, 3.73 [15:46:19] RECOVERY - wiki-tintin.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki-tintin.cf' will expire on Mon 10 Oct 2022 03:55:03 GMT +0000. [15:47:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:47:40] PROBLEM - abeyancewiki.bumblebee-studios.se - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.40, 9.45, 8.35 [15:49:31] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [15:50:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 4.10, 3.83 [15:50:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.36, 7.99, 7.96 [15:50:39] RECOVERY - lab.wikimimo.com - reverse DNS on sslhost is OK: SSL OK - lab.wikimimo.com reverse DNS resolves to cp31.miraheze.org - CNAME OK [15:51:01] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:21] RECOVERY - wiki.dereric.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dereric.gq' will expire on Sat 22 Oct 2022 22:30:02 GMT +0000. [15:51:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.02, 10.18, 9.66 [15:51:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.92, 3.47 [15:52:02] RECOVERY - zenith.nsmbu.net - LetsEncrypt on sslhost is OK: OK - Certificate 'zenith.nsmbu.net' will expire on Mon 10 Oct 2022 05:53:48 GMT +0000. [15:52:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.62, 3.48 [15:53:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.37, 10.38, 9.79 [15:54:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.63, 2.35, 1.55 [15:54:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 4.19, 3.72 [15:56:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 1.70, 1.23 [15:56:58] [02mediawiki] 07Universal-Omega labeled pull request 03#8313: Bump skins/Cosmos from `48e1b8a` to `c58ac13` - 13https://github.com/miraheze/mediawiki/pull/8313 [15:56:59] [url] Page not found · GitHub · GitHub | github.com [15:57:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.07, 9.08, 9.48 [15:58:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.92, 3.72 [15:59:53] Reception123: no, MacFan4000 fixed that, had to adjust the timeout like what he did for the rest of mw*. Also I’m not sure where cgroup sets it memory config as I cannot see it in puppet. I don’t see OOM in the grafana graph either [16:00:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.34, 5.21, 4.21 [16:02:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.46, 9.74, 8.24 [16:03:08] PROBLEM - celeste.ink - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.65, 1.80, 1.68 [16:04:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.74, 1.61, 1.47 [16:06:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.61, 1.37, 1.54 [16:06:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.53, 9.93, 8.74 [16:06:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:07:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.80, 3.99 [16:08:06] RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Tue 23 Aug 2022 07:53:50 GMT +0000. [16:08:06] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.73, 10.66, 9.65 [16:09:27] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 4.30, 4.16 [16:10:24] PROBLEM - lcn.zfc.id.lv - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:49] RECOVERY - www.mcpk.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mcpk.wiki' will expire on Sat 17 Sep 2022 18:44:27 GMT +0000. [16:11:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.56, 9.46, 9.32 [16:11:43] RECOVERY - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssangyongsports.eu.org' will expire on Wed 31 Aug 2022 22:12:11 GMT +0000. [16:13:23] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:13:26] RECOVERY - avas.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avas.wiki' will expire on Mon 10 Oct 2022 04:52:24 GMT +0000. [16:13:44] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.39, 9.33, 8.34 [16:14:32] RECOVERY - hoodwiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'hoodwiki.xyz' will expire on Wed 19 Oct 2022 06:37:38 GMT +0000. [16:15:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.69, 3.95 [16:16:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.08, 8.64, 8.23 [16:17:19] RECOVERY - abeyancewiki.bumblebee-studios.se - LetsEncrypt on sslhost is OK: OK - Certificate 'abeyancewiki.bumblebee-studios.se' will expire on Wed 24 Aug 2022 16:26:15 GMT +0000. [16:18:15] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:18:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.11, 9.44, 8.50 [16:18:46] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [16:20:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.50, 3.96 [16:20:17] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.060 second response time [16:20:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.86, 9.54, 8.66 [16:20:50] RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Tue 23 Aug 2022 03:02:35 GMT +0000. [16:20:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.57, 10.68, 9.53 [16:21:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.38, 3.69 [16:21:48] PROBLEM - lab.wikimimo.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lab.wikimimo.com could not be found [16:23:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.41, 3.65 [16:23:47] PROBLEM - zenith.nsmbu.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [16:24:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.85, 11.14, 9.97 [16:24:55] PROBLEM - wiki.yahyabd.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.63, 3.69 [16:26:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.82, 10.95, 8.81 [16:26:35] PROBLEM - rothwell-leeds.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 11.55, 10.26 [16:27:28] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Wed 24 Aug 2022 03:07:55 GMT +0000. [16:27:44] PROBLEM - wiki.sketch.town - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.68, 11.26, 9.19 [16:28:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.32, 11.20, 10.30 [16:29:06] PROBLEM - sbs.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sbs.wiki could not be found [16:29:21] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [16:29:41] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.00, 3.88, 3.84 [16:30:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.92, 9.53, 8.79 [16:31:33] PROBLEM - sbs.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address sbs.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:31:35] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [16:32:16] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:49] RECOVERY - celeste.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Sat 24 Sep 2022 13:42:28 GMT +0000. [16:32:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.52, 9.58, 8.64 [16:33:27] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [16:33:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [16:34:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.97, 3.91 [16:34:18] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.047 second response time [16:34:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.12, 12.09, 10.84 [16:34:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.48, 9.20, 8.60 [16:36:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.29, 4.43, 4.06 [16:36:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.28, 10.62, 10.46 [16:37:19] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:34] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [16:37:41] PROBLEM - cp21 Stunnel HTTP for mon111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:48] RECOVERY - wiki.seamly.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seamly.net' will expire on Tue 23 Aug 2022 07:40:05 GMT +0000. [16:37:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [16:38:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.30, 8.88, 9.85 [16:38:58] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Tue 23 Aug 2022 01:13:17 GMT +0000. [16:39:27] RECOVERY - lcn.zfc.id.lv - LetsEncrypt on sslhost is OK: OK - Certificate 'lcn.zfc.id.lv' will expire on Wed 31 Aug 2022 21:26:51 GMT +0000. [16:39:39] RECOVERY - cp21 Stunnel HTTP for mon111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 37919 bytes in 2.091 second response time [16:40:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.10, 10.53, 9.24 [16:42:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.95, 10.59, 9.42 [16:43:33] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 04 Oct 2022 18:56:56 GMT +0000. [16:43:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.75, 1.97 [16:43:41] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 24 Aug 2022 02:12:36 GMT +0000. [16:45:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.46, 8.89, 8.15 [16:46:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.21, 9.49, 9.30 [16:47:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.84, 9.41, 8.44 [16:47:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 1.87, 1.95 [16:48:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.68, 3.92 [16:49:12] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.51, 3.36, 3.92 [16:49:30] PROBLEM - squareenixwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [16:49:58] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.67, 10.11, 8.92 [16:50:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.46, 10.10, 9.34 [16:51:07] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [16:51:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.29, 3.47, 3.94 [16:52:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.59, 9.47, 8.83 [16:52:50] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [16:53:04] RECOVERY - zenith.nsmbu.net - LetsEncrypt on sslhost is OK: OK - Certificate 'zenith.nsmbu.net' will expire on Mon 10 Oct 2022 05:53:48 GMT +0000. [16:53:25] RECOVERY - wiki.yahyabd.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yahyabd.xyz' will expire on Sun 14 Aug 2022 20:06:30 GMT +0000. [16:53:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.42, 1.90, 1.98 [16:53:55] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 0.018 second response time [16:55:12] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [16:55:18] RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Sat 17 Sep 2022 18:56:39 GMT +0000. [16:56:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.99, 10.14, 9.82 [16:57:04] RECOVERY - wiki.sketch.town - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sketch.town' will expire on Thu 13 Oct 2022 04:34:16 GMT +0000. [16:57:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.91, 3.70, 3.79 [16:58:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 3.70, 3.70 [16:59:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.68, 3.78 [16:59:11] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [17:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.68, 3.69 [17:01:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 4.33, 4.02 [17:01:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.79, 4.17, 3.90 [17:01:15] PROBLEM - recaptime.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:02:06] PROBLEM - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:02:59] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:22] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.79, 1.83 [17:03:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.99, 9.90, 8.97 [17:04:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.82, 3.95, 3.78 [17:04:54] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [17:05:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:05:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.87, 1.85 [17:05:45] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.78, 8.20, 8.46 [17:08:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.92, 3.85 [17:09:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.17, 3.64, 3.91 [17:09:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.78, 3.88 [17:09:07] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:38] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.64, 3.45, 3.11 [17:09:48] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:10:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 4.02, 3.88 [17:10:32] PROBLEM - www.christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.78, 11.07, 9.74 [17:11:04] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 3.042 second response time [17:11:36] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.61, 3.17, 3.05 [17:11:45] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.232 second response time [17:11:46] PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.59, 3.74 [17:12:30] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:01] PROBLEM - hoseki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.75, 1.78 [17:13:56] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:14:27] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:15:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.93, 4.12, 3.94 [17:15:18] PROBLEM - cloudwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:22] PROBLEM - it.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.36, 9.59, 8.77 [17:15:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.73, 1.77 [17:15:54] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.48, 0.64, 0.71 [17:16:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.53, 4.22, 3.93 [17:16:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.08, 11.01, 10.31 [17:17:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.95, 3.91 [17:17:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.98, 3.93 [17:17:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.85, 2.02, 1.86 [17:18:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.71, 3.78 [17:18:11] PROBLEM - wiki.consid.vn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.28, 9.25, 9.74 [17:18:43] RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Sat 01 Oct 2022 04:20:30 GMT +0000. [17:18:52] PROBLEM - famedata.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.61, 3.79 [17:19:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.45, 9.24, 8.87 [17:20:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.12, 4.41, 4.01 [17:20:50] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:20:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.82, 4.57, 4.13 [17:20:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.64, 4.31, 4.01 [17:21:07] PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:21:10] PROBLEM - wiki.timewisely.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.98, 3.91 [17:22:31] PROBLEM - www.programming.red - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.98, 3.93 [17:23:16] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:24:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 4.24, 4.01 [17:24:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.62, 4.44, 4.09 [17:25:02] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:57] PROBLEM - commons.ballmedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:46] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:28:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:29:04] PROBLEM - www.cgradegames.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:48] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.028 second response time [17:30:02] RECOVERY - recaptime.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:30:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [17:31:03] PROBLEM - thenintendowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:06] PROBLEM - wiki.exputra.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:13] PROBLEM - civclassic.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.27, 9.65, 8.66 [17:31:46] RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Sat 01 Oct 2022 16:03:14 GMT +0000. [17:32:30] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:45] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [17:32:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.70, 3.95 [17:33:12] PROBLEM - es.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.27, 8.76, 8.46 [17:33:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.52, 1.28 [17:33:40] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.238 second response time [17:34:34] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:34:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.11, 10.12, 9.32 [17:34:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.55, 4.59, 4.26 [17:34:49] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:03] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:35:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.21, 1.83, 1.43 [17:35:47] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.397 second response time [17:36:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.199 second response time [17:37:03] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.032 second response time [17:37:06] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.056 second response time [17:37:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.96, 1.53 [17:37:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.51, 10.30, 8.63 [17:38:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.87, 10.21, 9.61 [17:39:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.29, 2.32, 1.71 [17:40:09] PROBLEM - test2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:09] PROBLEM - sims.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:15] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:32] RECOVERY - www.christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sat 17 Sep 2022 18:34:21 GMT +0000. [17:40:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.20, 9.64, 9.48 [17:40:46] RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Tue 23 Aug 2022 00:16:11 GMT +0000. [17:40:55] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.97, 2.04, 1.44 [17:41:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.76, 10.24, 9.09 [17:41:36] RECOVERY - hoseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hoseki.wiki' will expire on Wed 17 Aug 2022 08:43:28 GMT +0000. [17:42:10] PROBLEM - cp20 Stunnel HTTP for puppet111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:42:50] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.87, 1.93, 1.47 [17:43:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.77, 10.00, 8.83 [17:43:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.17, 8.46, 8.58 [17:44:17] RECOVERY - cp20 Stunnel HTTP for puppet111 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.031 second response time [17:44:32] PROBLEM - cloudwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cloudwiki.org' expires in 15 day(s) (Wed 10 Aug 2022 16:56:11 GMT +0000). [17:44:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.23, 11.43, 10.19 [17:44:46] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 2.12, 1.60 [17:45:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.75, 9.40, 8.74 [17:46:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.87, 11.45, 10.35 [17:46:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.40, 1.85, 1.56 [17:48:50] RECOVERY - famedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'famedata.org' will expire on Thu 29 Sep 2022 07:42:43 GMT +0000. [17:49:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.40, 10.88, 9.47 [17:49:54] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fbe7cda9c790...6909d3ceb1da [17:50:04] [url] Comparing fbe7cda9c790...6909d3ceb1da · miraheze/ssl · GitHub | github.com [17:50:05] [02miraheze/ssl] 07MirahezeSSLBot 036909d3c - Bot: Update SSL cert for cloudwiki.org [17:50:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:50:16] PROBLEM - squareenixwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.74, 3.99 [17:50:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.90, 9.09, 9.65 [17:50:50] RECOVERY - fc.songcontests.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Mon 22 Aug 2022 22:44:59 GMT +0000. [17:51:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 10.87, 9.64 [17:51:21] PROBLEM - wiki.meeusen.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:35] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Tue 23 Aug 2022 09:31:28 GMT +0000. [17:52:28] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:52:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 4.20, 4.13 [17:52:38] PROBLEM - wiki.recaptime.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 9.62, 3.66, 2.24 [17:53:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.69, 10.09, 9.51 [17:54:00] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Mon 22 Aug 2022 22:44:29 GMT +0000. [17:54:00] PROBLEM - cnt.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [17:54:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.23, 3.80 [17:54:42] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [17:55:38] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:55:44] RECOVERY - commons.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.ballmedia.org' will expire on Mon 10 Oct 2022 06:16:04 GMT +0000. [17:56:06] PROBLEM - hi.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:58:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 4.00, 3.97 [17:58:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.40, 10.66, 10.20 [17:58:43] RECOVERY - www.cgradegames.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.cgradegames.net' will expire on Sat 20 Aug 2022 20:03:51 GMT +0000. [17:58:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.25, 1.88, 1.87 [17:59:40] PROBLEM - scum-wiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:42] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:59:48] PROBLEM - squareenix.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:56] RECOVERY - wiki.exputra.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.exputra.com' will expire on Fri 16 Sep 2022 18:09:57 GMT +0000. [17:59:58] RECOVERY - civclassic.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:00:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [18:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.81, 3.90 [18:00:20] PROBLEM - wiki.newnormal.fans - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:31] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:02] RECOVERY - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Fri 16 Sep 2022 12:56:22 GMT +0000. [18:01:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.61, 9.83, 9.42 [18:01:12] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:01:14] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:01:14] PROBLEM - wikipanel.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:02:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.81, 4.30, 4.05 [18:02:17] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'es.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:18:15 GMT +0000. [18:02:25] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:02:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.24, 1.35, 1.64 [18:02:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.06, 10.13, 8.84 [18:03:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.32, 9.39, 9.32 [18:03:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.83, 1.58, 1.87 [18:03:27] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.298 second response time [18:04:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:04:12] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:04:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.73, 3.93 [18:04:29] PROBLEM - jp.popira.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:43] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.614 second response time [18:04:52] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Mon 12 Sep 2022 18:36:58 GMT +0000. [18:04:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.95, 10.80, 9.25 [18:05:11] PROBLEM - www.ethene.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:12] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:06:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.86, 4.78, 4.28 [18:06:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.81, 9.86, 9.08 [18:07:05] PROBLEM - www.zenbuddhism.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:09] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.032 second response time [18:07:35] PROBLEM - wiki.controlistes.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:19] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.231 second response time [18:08:36] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:08:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 9.19, 9.96 [18:08:54] RECOVERY - test2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:08:54] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:09:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.45, 10.33, 8.50 [18:09:38] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 17 Sep 2022 21:29:52 GMT +0000. [18:09:49] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [18:10:32] PROBLEM - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:36] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.048 second response time [18:10:42] PROBLEM - wiki.rtapp.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.23, 1.60 [18:11:06] PROBLEM - wikicompliance.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [18:11:40] PROBLEM - test3.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:42] PROBLEM - burnout.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:46] PROBLEM - proficientprofit.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:21] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 10.66, 9.75 [18:13:24] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.16, 9.64, 8.69 [18:13:56] PROBLEM - lgbtqia.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.97, 4.00 [18:14:46] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [18:14:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.16, 2.28, 1.99 [18:15:05] PROBLEM - wiki.lu9.io - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:13] RECOVERY - it.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'it.famepedia.org' will expire on Sat 08 Oct 2022 06:08:33 GMT +0000. [18:15:28] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 4 backends are down. mw101 mw102 mw111 mw122 [18:16:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.49, 10.12, 9.84 [18:17:30] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:37] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:03] RECOVERY - wiki.consid.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consid.vn' will expire on Mon 22 Aug 2022 23:09:56 GMT +0000. [18:18:23] PROBLEM - central.ballmedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.25, 1.70, 1.82 [18:18:54] PROBLEM - www.journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:21] PROBLEM - iol.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:22] RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Sat 01 Oct 2022 04:20:30 GMT +0000. [18:19:23] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:19:25] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [18:19:47] RECOVERY - wiki.timewisely.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.timewisely.net' will expire on Mon 22 Aug 2022 22:01:18 GMT +0000. [18:19:49] PROBLEM - dcmultiverse.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.73, 3.99, 3.93 [18:20:34] PROBLEM - gs.sidem.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:36] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:20:49] RECOVERY - wiki.meeusen.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Sat 17 Sep 2022 21:26:27 GMT +0000. [18:21:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [18:21:24] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.29, 3.52, 3.05 [18:21:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.38, 10.32, 8.77 [18:21:51] RECOVERY - wiki.recaptime.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.tk' will expire on Sun 14 Aug 2022 00:05:19 GMT +0000. [18:22:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.67, 3.81 [18:22:48] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [18:23:06] RECOVERY - cnt.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:23:22] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.63, 3.01, 2.91 [18:23:30] PROBLEM - zenith.nsmbu.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:37] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:23:55] PROBLEM - vault.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 3.84, 3.85 [18:24:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.45, 1.70 [18:25:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.94, 9.47, 8.76 [18:25:45] RECOVERY - hi.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.famepedia.org' will expire on Tue 18 Oct 2022 14:04:28 GMT +0000. [18:26:27] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:27:53] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.293 second response time [18:28:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.71, 10.22, 9.20 [18:28:26] RECOVERY - scum-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'scum-wiki.com' will expire on Sat 22 Oct 2022 05:35:09 GMT +0000. [18:28:38] RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:29:27] RECOVERY - wiki.newnormal.fans - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.newnormal.fans' will expire on Fri 14 Oct 2022 16:20:05 GMT +0000. [18:30:06] RECOVERY - wikipanel.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipanel.org' will expire on Sun 02 Oct 2022 04:51:38 GMT +0000. [18:31:10] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:15:36 GMT +0000. [18:32:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.76, 3.89 [18:32:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.62, 9.07, 9.03 [18:34:19] RECOVERY - www.ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Sat 24 Sep 2022 13:58:45 GMT +0000. [18:34:27] RECOVERY - jp.popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'jp.popira.info' will expire on Mon 10 Oct 2022 06:13:03 GMT +0000. [18:36:13] PROBLEM - celeste.ink - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:55] PROBLEM - www.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:05] PROBLEM - dkpedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:28] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:56] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:30] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:38:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 1.58, 1.54 [18:38:48] PROBLEM - az.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:48] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.41, 9.95, 8.91 [18:39:05] RECOVERY - wiki.pablojprieto.com.es - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pablojprieto.com.es' will expire on Tue 23 Aug 2022 09:39:26 GMT +0000. [18:39:42] RECOVERY - wiki.rtapp.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rtapp.tk' will expire on Sun 14 Aug 2022 00:07:41 GMT +0000. [18:40:29] RECOVERY - burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Fri 19 Aug 2022 05:01:19 GMT +0000. [18:40:38] RECOVERY - proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [18:40:47] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:40:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.45, 1.50 [18:41:05] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.221 second response time [18:41:25] RECOVERY - test3.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:41:42] PROBLEM - notes.mridulpm.in - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:56] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:42:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.54, 3.99 [18:42:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.39, 10.02, 8.70 [18:42:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.12, 11.54, 9.81 [18:43:06] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:12] PROBLEM - cloudwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:29] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Wed 31 Aug 2022 18:20:36 GMT +0000. [18:43:39] PROBLEM - unmade.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:40] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.82, 4.02 [18:44:29] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:29] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:46] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.06, 8.96, 8.47 [18:45:05] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.243 second response time [18:45:16] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:45:48] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:46:08] PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.70, 3.75 [18:46:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:46:21] RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Mon 22 Aug 2022 23:00:39 GMT +0000. [18:46:25] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.457 second response time [18:46:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.67, 3.94 [18:46:29] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.661 second response time [18:46:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.57, 10.93, 10.02 [18:47:00] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.385 second response time [18:47:10] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.024 second response time [18:47:16] PROBLEM - www.turtletown.ca - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:32] RECOVERY - central.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'central.ballmedia.org' will expire on Mon 10 Oct 2022 06:35:15 GMT +0000. [18:47:43] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 3 % full [18:47:50] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.028 second response time [18:48:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.70, 3.35, 3.62 [18:48:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.71, 3.43, 3.82 [18:48:20] RECOVERY - www.journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Wed 31 Aug 2022 18:13:33 GMT +0000. [18:49:17] RECOVERY - gs.sidem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gs.sidem.wiki' will expire on Tue 23 Aug 2022 13:09:38 GMT +0000. [18:49:51] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.945 second response time [18:50:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [18:50:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.65, 3.87, 3.91 [18:50:42] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.336 second response time [18:51:07] PROBLEM - evil.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:15] PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:52:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.35, 10.18, 10.00 [18:52:54] RECOVERY - zenith.nsmbu.net - LetsEncrypt on sslhost is OK: OK - Certificate 'zenith.nsmbu.net' will expire on Mon 10 Oct 2022 05:53:48 GMT +0000. [18:53:20] PROBLEM - files.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:23] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.33, 10.61, 9.38 [18:54:07] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:54:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.89, 3.87, 3.90 [18:55:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.10, 11.30, 9.77 [18:56:02] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.30, 0.80, 0.64 [18:56:35] PROBLEM - www.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:55] PROBLEM - www.portalsofphereon.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:30] PROBLEM - wiki.insideearth.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.47, 10.53, 8.81 [18:58:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.32, 3.45, 3.51 [18:58:12] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.78, 3.84 [18:58:16] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:58:40] RECOVERY - thenintendowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thenintendowiki.org' will expire on Sat 22 Oct 2022 05:06:23 GMT +0000. [18:58:45] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:59:12] PROBLEM - wiki.sketch.town - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.49, 10.24, 9.77 [18:59:35] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.16, 3.41 [19:00:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.80, 3.84 [19:01:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.42, 8.57, 9.21 [19:02:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.25, 9.67, 8.89 [19:02:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.61, 3.04, 3.34 [19:03:04] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [19:03:46] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.57, 9.61, 8.57 [19:04:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.13, 4.28, 4.00 [19:04:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.29, 4.85, 4.08 [19:04:50] PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:44] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.055 second response time [19:05:53] RECOVERY - celeste.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Sat 24 Sep 2022 13:42:28 GMT +0000. [19:06:09] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.15, 10.09, 8.89 [19:06:09] RECOVERY - wiki.controlistes.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.controlistes.fr' will expire on Sun 02 Oct 2022 05:16:52 GMT +0000. [19:06:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 4.30, 3.82 [19:06:32] !log [reception@test101] starting deploy of {'config': True, 'world': True, 'l10n': True} to all [19:06:34] PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:56] RECOVERY - www.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'www.wikiretet.ga' will expire on Tue 11 Oct 2022 04:42:43 GMT +0000. [19:07:08] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Thu 01 Sep 2022 00:46:11 GMT +0000. [19:07:20] paladox: the 503s aren't a MW issue since I just got one for Phab [19:07:24] so something is going on at another level [19:07:40] RECOVERY - az.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'az.wikiretet.ga' will expire on Tue 11 Oct 2022 04:40:37 GMT +0000. [19:07:41] but I can't tell what, it's only been happening for the past 7 days or so [19:08:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.85, 8.85, 7.64 [19:09:06] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:10:27] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:02] RECOVERY - wikicompliance.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikicompliance.com' will expire on Fri 07 Oct 2022 11:39:30 GMT +0000. [19:11:36] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.312 second response time [19:11:42] PROBLEM - iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:12:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.38, 3.56, 3.81 [19:12:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.93, 3.82 [19:12:25] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.033 second response time [19:12:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.93, 3.94 [19:12:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.05, 8.99, 8.10 [19:12:54] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:12:59] RECOVERY - unmade.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:13:15] RECOVERY - lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lgbtqia.wiki' will expire on Fri 07 Oct 2022 11:35:52 GMT +0000. [19:13:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [19:13:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.37, 9.25, 9.03 [19:14:27] PROBLEM - nottingham.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.28, 4.25, 4.05 [19:15:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 10.26, 9.43 [19:15:57] PROBLEM - hoodwiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:16:50] RECOVERY - www.turtletown.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'www.turtletown.ca' will expire on Tue 20 Sep 2022 20:30:39 GMT +0000. [19:17:22] RECOVERY - iol.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'iol.wiki' will expire on Sun 14 Aug 2022 15:50:21 GMT +0000. [19:17:24] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [19:17:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.16, 9.74, 9.37 [19:18:14] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:19:07] RECOVERY - dcmultiverse.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:20:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.02, 10.45, 9.41 [19:20:26] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [19:20:35] RECOVERY - evil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'evil.wiki' will expire on Wed 10 Aug 2022 22:13:25 GMT +0000. [19:20:42] RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Fri 30 Sep 2022 17:38:06 GMT +0000. [19:21:26] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw101 mw111 mw112 [19:21:37] PROBLEM - cp20 Stunnel HTTP for test101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:58] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.81, 10.79, 9.66 [19:22:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 3.97, 3.88 [19:22:05] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:07] RECOVERY - files.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'files.pornwiki.org' will expire on Sat 17 Sep 2022 19:10:31 GMT +0000. [19:22:08] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:13] PROBLEM - cp21 Stunnel HTTP for test101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:26] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.86, 2.05, 1.61 [19:23:06] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Wed 24 Aug 2022 03:08:19 GMT +0000. [19:23:23] RECOVERY - vault.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vault.agentisai.com' will expire on Thu 29 Sep 2022 08:34:24 GMT +0000. [19:24:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.14, 3.78, 3.67 [19:25:10] RECOVERY - www.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [19:26:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.00, 8.45, 9.05 [19:26:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.58, 3.61 [19:26:39] RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Sat 17 Sep 2022 20:30:32 GMT +0000. [19:27:19] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.926 second response time [19:27:26] RECOVERY - wiki.insideearth.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Tue 23 Aug 2022 08:05:34 GMT +0000. [19:27:42] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:27:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.96, 3.98 [19:29:22] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [19:29:25] PROBLEM - test101 PowerDNS Recursor on test101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:30:11] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.76, 3.29, 2.98 [19:30:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.95, 1.98, 1.95 [19:30:57] PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:27] JohnLewis: Reception123 just got a 503 for Phab [19:31:27] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:33] So it can't be a MW issue [19:31:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.54, 4.01, 3.96 [19:32:10] RhinosF1: I just said that before yeah, something strange is going on as I can't remember a 503 for Phab in ages [19:32:13] same for wikis before 7 days ago [19:32:31] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:32] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:32] Reception123: JohnLewis joined after [19:32:33] [discord] webmail.* regularly gives me 503s [19:32:38] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:38] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:40] Was repeating so he can see [19:32:56] I mean there's not really a lot we can do [19:33:01] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:14] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:15] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:16] could it be packet loss again? [19:33:32] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.805 second response time [19:33:34] Unlikely as it would be all internal [19:33:34] JohnLewis: what's the issue [19:33:55] Resource utilisation if I was to guess, could be hardware, could be many things [19:33:56] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:56] PROBLEM - test101 HTTPS on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:59] PROBLEM - test101 SSH on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:02] I also strangely got ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Broken pipe when trying to ssh to test101 [19:34:08] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.20, 3.31, 3.06 [19:34:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.85, 3.87, 3.67 [19:34:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.91, 2.08, 1.97 [19:34:27] RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Sat 17 Sep 2022 22:28:24 GMT +0000. [19:34:30] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.345 second response time [19:34:32] PROBLEM - test101 ferm_active on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:34:33] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.023 second response time [19:34:42] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.136 second response time [19:35:00] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:35:08] !log [reception@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:35:16] RECOVERY - www.zenbuddhism.info - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zenbuddhism.info' will expire on Sat 17 Sep 2022 20:26:18 GMT +0000. [19:35:30] RhinosF1: ^ regarding the above error: fetch died of signal 9 [19:35:34] !log [reception@test101] starting deploy of {'world': True, 'l10n': True} to all [19:35:51] RECOVERY - test101 HTTPS on test101 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 549 bytes in 0.027 second response time [19:35:55] RECOVERY - test101 SSH on test101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:35:56] Reception123: 9 is SIGKILL [19:36:02] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.428 second response time [19:36:15] So that means something killed it [19:36:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.64, 1.99, 1.96 [19:36:19] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:36:27] RECOVERY - test101 ferm_active on test101 is OK: OK ferm input default policy is set [19:36:33] Check kernel logs [19:36:38] maybe [19:36:39] [Mon Jul 25 19:34:21 2022] Out of memory: Killed process 1923878 (git) total-vm:2296136kB, anon-rss:861048kB, file-rss:0kB, shmem-rss:0kB, UID:33 pgtables:2680kB oom_score_adj:0 [19:36:39] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.550 second response time [19:37:01] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.745 second response time [19:37:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 10.21, 8.96 [19:37:16] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.291 second response time [19:37:18] RECOVERY - test101 PowerDNS Recursor on test101 is OK: DNS OK: 1.299 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [19:37:27] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.356 second response time [19:38:18] Reception123: well [19:38:28] I think that's clear [19:38:58] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 6.965 second response time [19:39:05] PROBLEM - miraheze.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:11] yeah, as you can see running again [19:39:27] Good [19:39:29] RECOVERY - notes.mridulpm.in - LetsEncrypt on sslhost is OK: OK - Certificate 'notes.mridulpm.in' will expire on Tue 20 Sep 2022 12:00:20 GMT +0000. [19:39:49] PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:24] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:27] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.448 second response time [19:40:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [19:40:34] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [19:40:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.71, 3.17, 1.99 [19:41:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.79, 10.33, 9.38 [19:41:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.91, 9.75, 8.44 [19:41:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [19:41:46] RECOVERY - cp21 Stunnel HTTP for test101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.068 second response time [19:42:18] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:42:22] PROBLEM - www.istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:28] RECOVERY - cp20 Stunnel HTTP for test101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.081 second response time [19:42:46] RECOVERY - cloudwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cloudwiki.org' will expire on Sun 23 Oct 2022 16:49:57 GMT +0000. [19:43:04] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:43:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.88, 8.31, 8.75 [19:43:07] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [19:43:34] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [19:43:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:43:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.82, 9.41, 8.50 [19:44:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.98, 3.90 [19:44:13] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.18, 1.38, 1.68 [19:44:22] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99038 bytes in 7.619 second response time [19:44:32] RECOVERY - hoodwiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'hoodwiki.xyz' will expire on Wed 19 Oct 2022 06:37:38 GMT +0000. [19:44:38] PROBLEM - unmade.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.87, 1.95, 1.77 [19:44:44] RECOVERY - wiki.lu9.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lu9.io' will expire on Tue 06 Sep 2022 20:59:54 GMT +0000. [19:45:02] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [19:45:39] RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Sun 14 Aug 2022 15:58:21 GMT +0000. [19:46:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 4.14, 3.97 [19:47:20] PROBLEM - abeyancewiki.bumblebee-studios.se - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:46] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.79, 9.49, 7.58 [19:48:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.79, 3.87 [19:48:19] PROBLEM - central.ballmedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.86, 1.43, 1.60 [19:49:21] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:49:43] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.905 second response time [19:49:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.29, 9.35, 7.77 [19:50:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.49, 4.29, 4.03 [19:50:29] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:27] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:44] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.66, 3.94 [19:52:38] PROBLEM - wiki.dereric.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:53:25] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.312 second response time [19:53:45] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.324 second response time [19:54:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.57, 9.89, 8.30 [19:54:12] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:16] PROBLEM - mabi.myomi.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:55] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:34] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.49, 0.76, 0.71 [19:55:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.44, 4.14, 4.03 [19:56:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.21, 8.93, 8.16 [19:56:13] PROBLEM - cp20 Stunnel HTTP for test101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:20] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:53] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [19:57:27] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:37] PROBLEM - cp21 Stunnel HTTP for test101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.78, 9.48, 8.34 [19:58:02] RECOVERY - wiki.sketch.town - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sketch.town' will expire on Thu 13 Oct 2022 04:34:16 GMT +0000. [19:58:12] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [19:58:23] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 8.442 second response time [19:58:32] RECOVERY - cp20 Stunnel HTTP for test101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 4.952 second response time [19:58:36] !log [reception@test101] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1381s [19:58:36] PROBLEM - replication-ops.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:00] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32404 bytes in 4.021 second response time [19:59:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.34, 9.99, 8.73 [19:59:25] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.323 second response time [19:59:37] PROBLEM - rct.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:43] Reception123: it finished this time [19:59:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.83, 9.34, 8.41 [19:59:46] yep [19:59:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.74, 10.22, 9.33 [19:59:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.49, 3.63, 3.88 [20:00:04] PROBLEM - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.78, 3.92 [20:01:07] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [20:01:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.20, 11.02, 9.26 [20:01:42] RECOVERY - cp21 Stunnel HTTP for test101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.058 second response time [20:01:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.76, 11.74, 9.99 [20:01:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.01, 10.07, 8.37 [20:03:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.43, 10.96, 9.88 [20:03:51] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.85, 9.67, 8.42 [20:04:38] RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Tue 04 Oct 2022 18:55:51 GMT +0000. [20:05:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.07, 10.77, 9.58 [20:05:54] RECOVERY - wiki.meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Sat 17 Sep 2022 22:20:19 GMT +0000. [20:06:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.21, 10.50, 9.06 [20:06:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.82, 3.86 [20:07:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.57, 9.73, 9.32 [20:07:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.90, 3.80, 3.84 [20:08:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.65, 10.99, 9.42 [20:08:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [20:08:24] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [20:08:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.89, 9.43, 7.97 [20:09:07] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:09:23] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Tue 23 Aug 2022 07:32:52 GMT +0000. [20:09:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 11.73, 10.59 [20:09:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.90, 1.64 [20:10:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.84, 9.52, 9.08 [20:10:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.90, 3.90 [20:10:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.44, 8.29, 7.73 [20:10:31] PROBLEM - wiki.rtapp.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.37, 10.43, 9.71 [20:11:50] RECOVERY - www.istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [20:11:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.99, 1.71 [20:12:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [20:12:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.21, 4.52, 4.09 [20:12:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.51, 8.98, 9.27 [20:12:58] PROBLEM - largedu.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:40] PROBLEM - cp21 Stunnel HTTP for puppet111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:13:46] RECOVERY - nottingham.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nottingham.wiki' will expire on Mon 26 Sep 2022 06:16:48 GMT +0000. [20:13:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.33, 9.13, 8.28 [20:13:51] RECOVERY - unmade.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [20:13:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.32, 3.80, 3.86 [20:13:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.88, 1.69 [20:14:21] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw121 [20:15:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.40, 10.44, 10.54 [20:15:40] RECOVERY - cp21 Stunnel HTTP for puppet111 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.031 second response time [20:16:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.95, 3.96 [20:16:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.62, 3.96 [20:16:57] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:59] RECOVERY - abeyancewiki.bumblebee-studios.se - LetsEncrypt on sslhost is OK: OK - Certificate 'abeyancewiki.bumblebee-studios.se' will expire on Wed 24 Aug 2022 16:26:15 GMT +0000. [20:17:27] RECOVERY - central.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'central.ballmedia.org' will expire on Mon 10 Oct 2022 06:35:15 GMT +0000. [20:17:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.28, 1.89, 1.73 [20:17:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.46, 3.68 [20:18:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 4.05, 3.99 [20:18:18] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [20:18:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.83, 9.62, 8.35 [20:18:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.81, 3.98 [20:19:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.66, 9.62, 10.14 [20:19:31] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.85, 2.33, 1.55 [20:19:44] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/2272c04b7f7b [20:19:45] [02miraheze/CreateWiki] 07Universal-Omega 032272c04 - CreateWikiJson: only decode when actually used [20:19:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.44, 9.64, 8.86 [20:19:47] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/CreateWiki [20:19:47] [url] Page not found · GitHub · GitHub | github.com [20:19:48] [02CreateWiki] 07Universal-Omega opened pull request 03#328: CreateWikiJson: only decode when actually used - 13https://github.com/miraheze/CreateWiki/pull/328 [20:19:49] ... [20:19:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.94, 1.77 [20:20:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.58, 9.13, 8.32 [20:20:51] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:01] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [20:21:04] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:21:26] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.19, 1.85, 1.46 [20:21:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.21, 2.26, 1.90 [20:21:51] RECOVERY - wiki.dereric.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dereric.gq' will expire on Sat 22 Oct 2022 22:30:02 GMT +0000. [20:23:22] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.02, 1.59, 1.41 [20:23:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.51, 10.77, 10.49 [20:23:25] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Wed 24 Aug 2022 03:18:35 GMT +0000. [20:23:39] RECOVERY - mabi.myomi.net - LetsEncrypt on sslhost is OK: OK - Certificate 'mabi.myomi.net' will expire on Tue 23 Aug 2022 13:11:15 GMT +0000. [20:26:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:26:11] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [20:26:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.58, 9.90, 9.25 [20:27:19] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.08, 9.46, 10.04 [20:27:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.92, 1.91 [20:28:08] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [20:28:14] RECOVERY - replication-ops.com - LetsEncrypt on sslhost is OK: OK - Certificate 'replication-ops.com' will expire on Tue 18 Oct 2022 15:32:39 GMT +0000. [20:28:15] RECOVERY - rct.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rct.wiki' will expire on Mon 22 Aug 2022 21:33:31 GMT +0000. [20:29:09] RECOVERY - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipedia.yogfront.ooo' will expire on Mon 22 Aug 2022 21:48:31 GMT +0000. [20:29:45] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.03, 2.25, 2.02 [20:30:09] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.28, 3.69, 3.17 [20:30:29] RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Fri 16 Sep 2022 17:59:19 GMT +0000. [20:31:00] miraheze/CreateWiki - Universal-Omega the build passed. [20:31:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 10.77, 10.44 [20:31:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.60, 10.74, 9.59 [20:32:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.12, 9.92, 9.55 [20:33:11] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.75, 9.89, 10.16 [20:33:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.45, 10.02, 9.47 [20:34:05] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.21, 3.74, 3.34 [20:36:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.49, 10.30, 9.83 [20:37:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.02, 10.84, 10.51 [20:38:01] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.77, 3.35, 3.28 [20:38:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.66, 10.04, 9.79 [20:39:18] RECOVERY - archive.stellurgists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.stellurgists.wiki' will expire on Tue 23 Aug 2022 00:33:03 GMT +0000. [20:39:31] RECOVERY - wiki.rtapp.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rtapp.tk' will expire on Sun 14 Aug 2022 00:07:41 GMT +0000. [20:42:38] RECOVERY - largedu.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'largedu.eu.org' will expire on Wed 17 Aug 2022 15:37:36 GMT +0000. [20:43:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [20:44:59] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [20:46:56] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [20:47:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.54, 1.77, 1.97 [20:48:07] RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Mon 22 Aug 2022 22:45:26 GMT +0000. [20:50:17] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [20:50:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 10.77, 10.51 [20:51:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.78, 1.92 [20:51:47] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.53, 3.48, 3.25 [20:52:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.76, 10.43, 10.43 [20:53:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.73, 1.88 [20:53:45] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.35, 3.31, 3.20 [20:56:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.86, 9.90, 10.18 [20:59:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.72, 1.24, 1.63 [21:07:02] I had a look on phab121 and it's never given a 502 or a 503 in the past 24 hours. It's given some 500s but they were legit [21:07:08] RhinosF1: Reception123 ^ [21:07:19] So phab's looking healthy server side [21:07:55] (and paladox) [21:08:17] probably on the cache proxy [21:08:32] Yeah, which is directly a result of mw* being unavailable [21:08:40] I could change that behaviour I guess [21:08:41] RECOVERY - miraheze.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ml' will expire on Tue 23 Aug 2022 02:37:13 GMT +0000. [21:12:00] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2f7b43106dba...9a4073d831b5 [21:12:00] [url] Comparing 2f7b43106dba...9a4073d831b5 · miraheze/puppet · GitHub | github.com [21:12:01] [02miraheze/puppet] 07JohnFLewis 039a4073d - varnish: always return 200 on healthcheck from GDNSD [21:12:56] JohnLewis: I don't think checking on mw to depool a cp makes sense [21:13:13] That's always been the case [21:13:30] I just did that to prevent the request going through to mw* [21:13:57] Ye your commit seems fine JohnLewis [21:14:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 9.56, 8.40 [21:14:10] I guess in past mw was only real thing behind [21:14:18] But that's not true anymore [21:14:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.12, 10.34, 9.45 [21:14:45] And if mw is down, it's likely down from all cp* unless it's a cp issue which would break the return 200 anyway [21:14:58] Yeah [21:16:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.11, 10.22, 8.80 [21:16:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.05, 10.01, 9.45 [21:18:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.02, 9.72, 8.79 [21:26:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 10.60, 10.18 [21:30:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.21, 10.48, 9.94 [21:30:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.67, 11.31, 10.55 [21:30:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.83, 2.19, 1.65 [21:31:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/3eb54757be29 [21:31:19] [02miraheze/puppet] 07paladox 033eb5475 - varnish: force thumb_handler through a specific set of servers [21:31:20] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [21:31:21] [url] Page not found · GitHub · GitHub | github.com [21:31:22] [02puppet] 07paladox opened pull request 03#2703: varnish: force thumb_handler through a specific set of servers - 13https://github.com/miraheze/puppet/pull/2703 [21:31:22] ... [21:32:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3eb54757be29...8bd317720127 [21:32:16] [url] Comparing 3eb54757be29...8bd317720127 · miraheze/puppet · GitHub | github.com [21:32:17] [02miraheze/puppet] 07paladox 038bd3177 - Update common.yaml [21:32:18] [02puppet] 07paladox synchronize pull request 03#2703: varnish: force thumb_handler through a specific set of servers - 13https://github.com/miraheze/puppet/pull/2703 [21:32:19] [url] Page not found · GitHub · GitHub | github.com [21:32:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.17, 11.35, 10.33 [21:32:54] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.86, 1.76, 1.55 [21:34:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.57, 1.51 [21:36:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.92, 10.83, 10.41 [21:37:58] !log apply https://github.com/miraheze/puppet/pull/2703 on puppetserver [21:37:58] [url] varnish: force thumb_handler through a specific set of servers by paladox · Pull Request #2703 · miraheze/puppet · GitHub | github.com [21:38:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.29, 11.68, 10.78 [21:40:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.14, 10.76, 9.72 [21:40:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.94, 10.41, 10.44 [21:41:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.75, 10.92, 9.88 [21:42:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [21:42:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.63, 9.70, 10.19 [21:43:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.44, 11.13, 10.08 [21:44:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.08, 11.34, 10.19 [21:45:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.31, 11.09, 9.65 [21:46:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.68, 11.67, 10.46 [21:46:46] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:48] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:54] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:20] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.76, 11.68, 10.51 [21:48:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.20, 11.84, 10.67 [21:49:20] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.869 second response time [21:50:29] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:57] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.477 second response time [21:50:59] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.699 second response time [21:51:01] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.204 second response time [21:51:24] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:27] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:30] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.12, 11.55, 10.86 [21:52:34] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.812 second response time [21:53:28] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.535 second response time [21:53:31] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.132 second response time [21:53:31] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.789 second response time [21:53:39] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:53:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.14, 3.26, 3.94 [21:54:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.34, 2.93, 3.80 [21:54:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.79, 12.08, 11.16 [21:54:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 2.98, 3.84 [21:55:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.12, 11.15, 10.89 [21:55:10] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:11] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.019 second response time [21:55:23] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.66, 4.00 [21:56:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.59, 3.94 [21:56:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.59, 3.96 [21:58:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.48, 11.46, 11.09 [21:59:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.06, 11.92, 11.40 [22:03:56] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.121 second response time [22:03:59] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.987 second response time [22:04:04] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.827 second response time [22:05:11] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:28] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:33] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.54, 9.15, 10.19 [22:06:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.69, 8.71, 9.96 [22:06:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.37, 8.47, 6.62 [22:06:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.91, 10.04, 11.50 [22:06:58] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:24] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:29] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 3.924 second response time [22:07:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.46, 1.46, 1.99 [22:07:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.22, 9.75, 6.77 [22:07:51] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:19] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:20] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:22] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.58, 8.63, 10.09 [22:09:39] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.971 second response time [22:09:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.39, 9.75, 7.12 [22:09:48] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.126 second response time [22:09:49] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 8.531 second response time [22:09:57] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.743 second response time [22:10:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.32, 8.50, 7.17 [22:10:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.59, 10.86, 11.46 [22:11:04] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.984 second response time [22:11:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.22, 8.35, 6.93 [22:12:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.14, 9.79, 10.97 [22:13:38] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.332 second response time [22:13:41] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.814 second response time [22:13:42] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [22:13:59] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [22:14:24] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:26] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.612 second response time [22:14:34] !log reverted [22:14:38] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.136 second response time [22:14:42] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.607 second response time [22:14:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 16.62, 12.46, 11.81 [22:15:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:39] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.325 second response time [22:15:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.55, 1.31 [22:15:52] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [22:16:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.72, 11.44, 10.57 [22:16:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.78, 12.30, 10.95 [22:16:19] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.313 second response time [22:16:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.80, 11.43, 11.53 [22:17:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.60, 1.85, 1.86 [22:17:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.84, 1.37, 1.27 [22:17:45] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.36, 9.21, 9.86 [22:18:01] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.52, 10.26, 10.38 [22:18:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.81, 12.06, 11.73 [22:19:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.99, 9.08, 7.54 [22:19:44] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.037 second response time [22:20:02] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.722 second response time [22:20:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 16.20, 12.70, 11.26 [22:21:28] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.64, 10.14, 8.11 [22:21:36] PROBLEM - evil.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'evil.wiki' expires in 15 day(s) (Wed 10 Aug 2022 22:13:25 GMT +0000). [22:23:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6909d3ceb1da...aa627355d680 [22:23:23] [url] Comparing 6909d3ceb1da...aa627355d680 · miraheze/ssl · GitHub | github.com [22:23:24] [02miraheze/ssl] 07MirahezeSSLBot 03aa62735 - Bot: Update SSL cert for evil.wiki [22:23:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.29, 10.00, 8.27 [22:23:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 2.05, 1.59 [22:25:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.14, 10.23, 8.54 [22:25:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.42, 1.81, 1.55 [22:27:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.68, 9.95, 8.66 [22:27:32] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.30, 2.49, 1.82 [22:28:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.94, 11.44, 11.57 [22:29:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.36, 1.82, 2.00 [22:30:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 16.22, 12.60, 11.93 [22:33:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.76, 1.71 [22:33:51] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.018 second response time [22:34:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.93, 11.02, 11.48 [22:34:09] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.19, 6.38, 3.86 [22:34:16] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:20] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:32] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:52] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.45, 1.60 [22:36:09] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.26, 6.73, 4.31 [22:36:55] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.991 second response time [22:37:48] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [22:38:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.93, 11.91, 11.68 [22:38:09] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.81, 6.03, 4.35 [22:38:17] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.034 second response time [22:38:21] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.606 second response time [22:38:41] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [22:39:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.80, 1.87 [22:40:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.12, 13.49, 11.13 [22:42:13] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:42:48] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:43:07] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:43:24] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:44:50] PROBLEM - ldap111 Puppet on ldap111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:45:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.62, 1.57 [22:45:21] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:46:10] PROBLEM - es121 Puppet on es121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:47:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.93, 1.37, 1.48 [22:47:26] PROBLEM - gluster101 Puppet on gluster101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:48:44] PROBLEM - es101 Puppet on es101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:48:49] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:50:12] hmm [22:50:16] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:50:45] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:51:03] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:51:16] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.58, 1.53 [22:51:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.98, 1.99 [22:51:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.02, 10.81, 9.44 [22:51:48] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:51:59] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:01] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:04] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:12] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:26] PROBLEM - mem101 Puppet on mem101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:43] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:48] PROBLEM - prometheus101 Puppet on prometheus101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:52:50] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:53:01] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:53:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.62, 1.87, 1.64 [22:53:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 2.03, 2.01 [22:53:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.92, 10.84, 9.60 [22:54:10] PROBLEM - es111 Puppet on es111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:54:34] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:55:13] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.84, 1.66 [22:55:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.99, 2.00 [22:55:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.36, 11.18, 9.87 [22:56:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.96, 11.65, 11.71 [22:56:10] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:57:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.68, 1.63 [22:57:33] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:57:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.17, 2.08, 2.03 [22:57:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.42, 10.72, 9.85 [22:58:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.69, 12.64, 12.05 [22:58:57] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:00:44] PROBLEM - gluster111 Puppet on gluster111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:00:52] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:01:00] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:01:08] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:01:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.73, 1.68 [23:01:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.99, 2.00 [23:01:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.23, 9.87, 9.71 [23:02:47] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.27, 1.55, 1.61 [23:03:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 2.07, 2.02 [23:04:26] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:04:43] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:05:49] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:05:59] PROBLEM - bast101 Puppet on bast101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:07:29] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:09:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.97, 1.99 [23:09:40] PROBLEM - db111 Puppet on db111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:11:08] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:11:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 2.02, 2.01 [23:12:04] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:12:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [23:13:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.98, 2.00 [23:15:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 2.04, 2.02 [23:22:50] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:01] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:04] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:24:10] RECOVERY - es111 Puppet on es111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.90, 11.60, 10.49 [23:24:12] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:34] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:24:43] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:25:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:25:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.90, 1.99 [23:26:10] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:28:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.66, 11.12, 10.56 [23:28:44] RECOVERY - gluster111 Puppet on gluster111 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:28:52] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:28:58] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:29:00] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:08] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:31:11] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.41, 3.25, 2.86 [23:31:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.97, 1.98 [23:31:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.81, 9.75, 10.17 [23:32:26] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:32:43] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:47] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:33:09] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.38, 3.02, 2.83 [23:33:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.51, 1.83, 1.93 [23:33:59] RECOVERY - bast101 Puppet on bast101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:35:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.94, 1.96 [23:35:49] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:29] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:37:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.80, 1.90 [23:39:07] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:39:08] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:39:40] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:40:04] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:40:48] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:40:50] RECOVERY - ldap111 Puppet on ldap111 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:42:13] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:43:22] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:43:24] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:10] RECOVERY - es121 Puppet on es121 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:45:26] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:38] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.39, 1.42, 1.67 [23:46:49] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:43] RECOVERY - es101 Puppet on es101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.78, 2.00, 1.83 [23:49:49] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:50:00] RECOVERY - evil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'evil.wiki' will expire on Sun 23 Oct 2022 21:23:17 GMT +0000. [23:50:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.36, 9.29, 11.93 [23:50:01] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:27] RECOVERY - mem101 Puppet on mem101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:49] RECOVERY - prometheus101 Puppet on prometheus101 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:51:01] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:59] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:16] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:52:45] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:53:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.08, 9.38, 11.65 [23:55:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.10, 2.32, 1.58 [23:55:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.41, 1.85, 1.86 [23:58:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.04, 7.46, 9.91 [23:59:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.58, 1.86, 1.55 [23:59:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.83, 1.83