[00:00:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.55, 10.40, 9.76 [00:02:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.10, 4.28, 3.93 [00:02:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.59, 3.46 [00:02:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.42, 10.05, 9.71 [00:04:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.82, 3.81 [00:06:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.01, 4.77, 4.16 [00:13:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.83, 3.77 [00:19:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.71, 3.68 [00:19:37] RECOVERY - wiki.recaptime.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.eu.org' will expire on Mon 03 Oct 2022 22:23:06 GMT +0000. [00:21:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.57, 6.62, 6.06 [00:21:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.05, 10.19, 9.87 [00:21:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.35, 3.55 [00:21:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.30, 1.60, 1.44 [00:23:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.64, 10.00, 9.84 [00:23:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.45, 1.43, 1.39 [00:25:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 11.64, 8.52, 6.89 [00:26:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.76, 4.00 [00:30:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.88, 3.98 [00:31:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.02, 7.84, 7.20 [00:31:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 2.98, 3.36 [00:32:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.35, 3.77 [00:33:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 10.28, 10.02 [00:37:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.80, 6.13, 6.67 [00:38:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.71, 3.39, 3.95 [00:39:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.74, 10.21, 9.39 [00:40:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.07, 2.90, 3.40 [00:41:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.96, 9.88, 9.38 [00:42:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.14, 8.65, 9.67 [00:44:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.16, 3.77, 3.63 [00:44:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.62, 3.71, 3.89 [00:46:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.63, 3.59 [00:46:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 3.92, 3.55 [00:46:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.62, 3.82 [00:48:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.07, 3.76 [00:48:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.24, 4.18, 4.00 [00:50:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.97, 3.76 [00:50:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.92, 3.65 [00:50:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.40, 3.75, 3.86 [00:52:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.72, 1.56 [00:56:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.66, 3.63 [00:56:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.54, 1.54 [00:58:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.55, 3.59 [00:59:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.02, 3.34 [01:00:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.21, 3.82, 3.84 [01:02:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.39, 3.27, 3.63 [01:06:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.48, 1.25 [01:08:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.64, 1.72, 1.63 [01:08:24] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.20, 1.39, 1.25 [01:08:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 2.79, 3.32 [01:09:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.60, 1.65, 1.62 [01:11:20] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:12:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.40, 3.24 [01:12:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.40, 3.24 [01:14:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 3.77, 3.48 [01:14:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.59, 3.34 [01:18:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 3.78, 3.43 [01:18:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.99, 3.65 [01:18:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.80, 1.69 [01:19:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.80, 3.56 [01:20:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 3.36, 3.30 [01:20:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 4.38, 3.84 [01:20:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.31, 1.94, 1.76 [01:22:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.96, 1.79 [01:24:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.66, 1.70 [01:27:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.67, 3.58 [01:28:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.19, 10.13, 8.69 [01:30:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.34, 9.44, 8.61 [01:32:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.24, 10.49, 9.41 [01:33:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.11, 3.39, 3.56 [01:34:02] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.93, 9.90, 9.33 [01:34:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 4.00, 3.99 [01:37:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.68, 3.66 [01:39:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.58, 3.63 [01:39:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.42, 3.30 [01:40:09] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [01:41:36] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.44, 1.63, 1.52 [01:43:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.84, 9.85 [01:43:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.16, 3.37, 3.31 [01:43:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.38, 1.50, 1.49 [01:44:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 4.13, 4.02 [01:45:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.23, 10.96, 10.02 [01:46:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.83, 3.93 [01:47:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.64, 1.83, 1.66 [01:49:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.34, 1.68, 1.62 [01:52:42] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.13, 2.22, 1.73 [01:52:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.76, 3.73, 3.59 [01:53:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.34, 11.13, 10.35 [01:54:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.49, 3.52 [01:55:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.53, 10.74, 10.30 [01:56:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 3.71, 3.59 [01:59:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.87, 10.19, 10.20 [02:00:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.37, 4.05, 3.77 [02:02:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.05, 1.71, 1.77 [02:03:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.48, 3.37 [02:04:24] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.17, 1.52, 1.69 [02:06:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 10.63, 10.33 [02:08:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.21, 3.54, 3.80 [02:09:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.05, 3.20, 3.29 [02:10:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.95, 4.00 [02:10:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.66, 3.79 [02:12:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.80, 4.24, 4.12 [02:13:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.59, 3.42 [02:14:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.76, 3.95 [02:14:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.55, 3.46, 3.74 [02:14:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.69, 9.76, 10.09 [02:15:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.16, 3.00, 3.22 [02:15:52] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.27, 6.68, 5.99 [02:18:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 3.35, 3.58 [02:19:43] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.69, 6.77, 6.19 [02:20:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.09, 3.45 [02:22:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 2.00, 1.65, 1.47 [02:22:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.55, 2.82, 3.30 [02:24:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.16, 1.83, 1.56 [02:24:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.16, 3.44 [02:25:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.23, 1.61, 1.51 [02:26:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 2.92, 3.32 [02:36:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.99, 10.69, 10.12 [02:38:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.24, 4.41, 3.54 [02:39:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.51, 3.09 [02:39:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.99, 1.86, 1.60 [02:40:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.62, 3.28 [02:40:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.59, 9.64, 9.83 [02:41:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 2.08, 1.72 [02:42:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.95, 3.54 [02:43:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.81, 1.67 [02:45:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 3.22, 3.14 [02:45:36] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.51, 2.42, 1.89 [02:46:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.93, 3.13 [02:46:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.19, 3.32 [02:51:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.98, 1.89 [02:53:13] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.62, 1.76, 1.50 [02:55:07] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.76, 1.47, 1.43 [02:56:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 3.48, 3.22 [02:58:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.93, 3.13, 3.11 [03:00:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.82, 3.47 [03:00:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.63, 2.07, 1.74 [03:01:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.78, 1.24, 1.60 [03:02:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.74, 1.59, 1.60 [03:02:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.01, 3.26, 3.31 [03:04:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.72, 3.36 [03:05:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.65, 3.27 [03:06:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.69, 3.39 [03:06:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.64, 3.50 [03:07:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 3.29, 3.18 [03:08:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 3.94, 3.51 [03:11:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.56, 3.34 [03:11:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.61, 9.96, 9.04 [03:12:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.91, 3.63 [03:12:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.05, 3.84, 2.51 [03:13:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.27, 3.25 [03:13:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.61, 9.26, 8.89 [03:14:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.13, 3.35 [03:16:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.38, 4.31, 3.83 [03:18:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.85, 3.71 [03:21:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.26, 9.79, 9.43 [03:22:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.60, 9.81, 9.08 [03:22:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.09, 1.77, 2.00 [03:24:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.13, 1.82, 1.98 [03:25:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.52, 9.71, 9.48 [03:25:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.85, 10.19, 9.32 [03:26:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.61, 9.99, 9.34 [03:26:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.00, 3.06, 3.36 [03:26:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.35, 1.69, 1.92 [03:27:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.64, 9.72, 9.23 [03:28:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.44, 3.36 [03:29:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.02, 10.50, 9.85 [03:32:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.43, 2.27, 2.05 [03:32:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.98, 3.32, 3.35 [03:33:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.51, 9.95, 9.79 [03:34:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.71, 1.95, 1.95 [03:36:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.44, 3.40 [03:36:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.69, 3.37 [03:36:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.49, 10.57, 10.10 [03:38:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.29, 4.29, 3.63 [03:38:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.81, 3.54 [03:38:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.66, 3.51 [03:38:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.02, 10.31 [03:40:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.96, 3.60 [03:40:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.58, 3.49 [03:40:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.78, 10.41, 10.20 [03:42:24] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.78, 1.39, 1.70 [03:42:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.38, 3.07, 3.29 [03:45:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.70, 6.47, 5.91 [03:45:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.38, 3.11, 3.36 [03:46:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.16, 3.32 [03:47:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.73, 6.41, 5.95 [03:50:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.94, 1.59, 1.64 [03:53:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.08, 6.93, 6.32 [03:54:24] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.23, 1.48, 1.59 [04:01:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.48, 6.39, 6.37 [04:01:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.12, 9.99, 9.38 [04:03:44] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.78, 9.81, 9.39 [04:04:24] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.62, 11.20, 10.79 [04:06:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.09, 11.19, 10.85 [04:08:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 11.24, 10.89 [04:10:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.99, 9.75, 10.39 [04:14:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.28, 9.28, 10.09 [04:15:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.56, 3.08 [04:17:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.62, 3.09 [04:19:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.98, 3.40, 3.07 [04:24:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.99, 10.42, 10.15 [04:25:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 3.27, 3.29 [04:29:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.91, 4.18, 3.46 [04:30:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.33, 10.11, 10.13 [04:32:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.69, 1.39 [04:32:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.77, 3.36 [04:32:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.79, 3.48 [04:33:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.82, 3.81, 3.57 [04:34:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.53, 1.38 [04:34:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.72, 3.03, 3.14 [04:34:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.14, 3.38, 3.36 [04:36:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.96, 10.48, 10.27 [04:37:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.90, 3.65 [04:38:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.84, 9.95, 10.08 [04:39:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.50, 3.53 [04:43:18] PROBLEM - wiki.wallymer.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wallymer.com could not be found [04:43:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.69, 3.57 [04:45:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.65, 3.43 [04:45:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.59, 3.55 [04:46:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.49, 3.43 [04:47:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 3.15, 3.27 [04:48:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 3.16, 3.31 [04:49:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.95, 3.15, 3.37 [05:00:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.95, 9.81, 9.07 [05:02:01] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.79, 6.94, 6.15 [05:02:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.68, 9.38, 9.01 [05:03:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.59, 10.46, 9.27 [05:03:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.02, 6.79, 6.20 [05:05:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.78, 9.98, 9.23 [05:07:49] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.42, 6.99, 6.43 [05:09:45] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.68, 6.52, 6.32 [05:13:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 10.58, 9.73 [05:13:38] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.74, 7.04, 6.56 [05:15:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.26, 9.68, 8.85 [05:15:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.71, 10.80, 9.78 [05:17:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.66, 9.79, 9.00 [05:18:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.89, 11.04, 10.17 [05:19:25] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.64, 9.79, 9.63 [05:21:20] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.13, 7.69, 7.09 [05:22:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.91, 10.08, 9.99 [05:22:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.74, 10.18, 10.00 [05:23:16] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.48, 7.57, 7.12 [05:26:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.44, 10.94, 10.36 [05:27:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.67, 7.90, 7.32 [05:32:57] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.76, 11.23, 10.39 [05:34:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.21, 10.89, 10.38 [05:36:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 2.99, 2.79 [05:36:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 11.67, 11.00 [05:38:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.24, 2.94, 2.79 [05:38:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.22, 11.16, 10.89 [05:42:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.44, 3.05 [05:43:10] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.73, 11.10, 10.04 [05:44:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.75, 3.22, 3.02 [05:45:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.59, 10.82, 10.09 [05:47:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.13, 9.51, 9.69 [05:48:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.36, 3.10 [05:50:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.99, 3.01, 3.01 [05:50:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.00, 9.15, 9.93 [05:54:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.98, 9.50, 10.09 [05:59:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.80, 7.23, 8.00 [06:01:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.30, 7.53, 8.00 [06:02:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.54, 3.30 [06:03:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.99, 7.37, 7.89 [06:04:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.22, 3.22, 3.19 [06:08:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.68, 3.96, 3.36 [06:09:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.64, 3.38 [06:10:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.56, 3.28 [06:11:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.82, 3.47 [06:12:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.19, 10.28, 9.29 [06:13:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.94, 10.23, 9.82 [06:13:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.43, 3.37 [06:13:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.17, 3.19 [06:14:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.75, 9.65, 9.17 [06:15:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.27, 9.18, 9.48 [06:15:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.70, 3.17, 3.28 [06:18:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 10.21, 9.84 [06:20:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.93, 9.49, 9.61 [06:21:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.47, 7.78, 7.45 [06:22:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.81, 10.62, 9.73 [06:23:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.45, 7.57, 7.42 [06:24:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.39, 10.64, 9.91 [06:24:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.01, 11.08, 10.24 [06:27:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.20, 7.51, 7.39 [06:27:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.72, 3.28 [06:28:27] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.80, 10.13, 9.80 [06:29:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.98, 7.79, 7.52 [06:29:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.05, 3.06, 3.09 [06:30:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.56, 10.92, 10.51 [06:32:03] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [06:32:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.60, 11.15, 10.25 [06:33:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.24, 7.73, 7.52 [06:34:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.34, 10.98, 10.31 [06:34:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 11.38, 10.77 [06:35:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.63, 7.62, 7.51 [06:36:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.56, 10.08, 10.14 [06:37:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.33, 7.63, 7.51 [06:38:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.09, 11.34, 10.91 [06:40:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.85, 10.01, 10.09 [06:40:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.31, 11.05, 10.49 [06:44:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 10.63, 10.52 [06:45:27] PROBLEM - matomo101 NTP time on matomo101 is WARNING: NTP WARNING: Offset -0.1029033363 secs [06:46:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.59, 11.07, 10.50 [06:46:24] PROBLEM - mw111 NTP time on mw111 is WARNING: NTP WARNING: Offset -0.1042025089 secs [06:52:03] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [06:52:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.37, 9.91, 10.15 [06:52:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.02, 9.78, 10.17 [06:52:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.42, 9.88, 9.23 [06:53:12] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.73, 3.60, 2.51 [06:54:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.82, 11.25, 10.90 [06:56:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.71, 11.22, 10.93 [06:56:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.18, 9.66, 9.30 [06:57:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.89, 7.13, 7.89 [06:57:12] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.97, 3.86, 2.88 [06:58:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.91, 11.61, 11.09 [07:00:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.57, 11.77, 11.23 [07:01:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.33, 7.70, 7.93 [07:01:12] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.82, 3.29, 2.88 [07:05:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.67, 7.34, 7.74 [07:08:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.03, 12.15, 11.48 [07:10:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.89, 11.82, 11.44 [07:11:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.48, 7.80, 7.81 [07:14:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 11.73, 11.47 [07:17:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.06, 7.91, 7.90 [07:18:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 11.74, 11.55 [07:19:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.51, 8.33, 8.05 [07:20:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 11.92, 11.64 [07:22:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.48, 11.51, 11.53 [07:27:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.96, 9.91, 9.60 [07:28:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.46, 11.63, 11.52 [07:29:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.17, 2.57 [07:29:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.08, 10.80, 9.94 [07:30:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.67, 11.37, 11.43 [07:31:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 2.86, 2.52 [07:33:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.59, 11.10, 10.34 [07:38:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.99, 10.73, 9.64 [07:39:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.64, 10.09, 10.18 [07:41:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.93, 11.20, 10.02 [07:43:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.50, 10.33, 9.86 [07:44:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.40, 8.97, 10.15 [07:45:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.79, 9.54, 9.63 [07:47:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.57, 7.48, 7.98 [07:52:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.48, 10.57, 10.16 [07:53:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.47, 2.01, 1.39 [07:55:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.72, 1.58, 1.31 [07:56:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 10.47, 10.24 [07:59:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.07, 9.47, 9.20 [08:00:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.73, 9.49, 9.99 [08:01:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.09, 10.99, 9.81 [08:03:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.63, 10.90, 9.93 [08:05:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.64, 7.40, 7.26 [08:06:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.40, 10.25, 9.81 [08:07:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.32, 7.19, 7.19 [08:07:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.13, 9.95, 9.76 [08:08:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.82, 10.17, 9.84 [08:21:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.21, 7.15, 7.03 [08:23:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.23, 2.81 [08:24:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 10.50, 9.87 [08:25:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.17, 3.09, 2.81 [08:26:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.43, 10.08, 9.79 [08:27:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.24, 7.64, 7.36 [08:28:18] RECOVERY - mw111 NTP time on mw111 is OK: NTP OK: Offset -0.09914731979 secs [08:29:21] RECOVERY - matomo101 NTP time on matomo101 is OK: NTP OK: Offset -0.09700480103 secs [08:31:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.17, 8.05, 7.55 [08:33:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.60, 7.78, 7.51 [08:35:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.03, 7.98, 7.61 [08:37:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.72, 7.54, 7.50 [08:39:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.13, 7.84, 7.62 [08:41:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.14, 7.54, 7.53 [08:43:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.63, 7.78, 7.61 [08:46:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.57, 9.99, 9.68 [08:48:27] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.03, 9.57, 9.56 [08:49:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.67, 7.85, 7.71 [08:51:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.89, 8.20, 7.85 [08:52:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.26, 10.85, 10.07 [08:52:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.87, 10.11, 9.12 [08:54:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.83, 10.82, 10.16 [08:54:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.82, 9.39, 8.99 [08:54:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.50, 4.08, 3.17 [08:54:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.34, 3.64, 2.98 [08:54:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.28, 3.78, 2.92 [08:55:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.99, 7.98, 7.87 [08:55:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.66, 1.71, 1.19 [08:57:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.36, 8.41, 8.03 [08:57:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.28, 1.61, 1.22 [08:58:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.01, 10.95, 10.34 [09:00:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.62, 10.37, 10.20 [09:02:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.68, 3.46 [09:03:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.49, 10.64, 10.07 [09:04:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 3.40, 3.38 [09:06:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.97, 3.79 [09:07:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.16, 11.27, 10.43 [09:08:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.38, 4.14, 3.65 [09:09:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.21, 11.13, 10.50 [09:10:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.49, 3.47 [09:10:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.09, 3.42, 3.64 [09:11:20] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:12:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 3.20, 3.37 [09:13:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.98, 9.29, 9.90 [09:13:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.88, 3.63, 3.61 [09:14:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.63, 3.65 [09:16:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.25, 9.50, 9.98 [09:17:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.65, 3.61 [09:18:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.80, 3.60 [09:18:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.06, 3.42, 3.62 [09:20:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.97, 3.67 [09:22:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.48, 3.53 [09:23:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.10, 2.81, 3.26 [09:25:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.87, 10.17, 10.04 [09:25:47] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [09:25:48] [url] Page not found · GitHub · GitHub | github.com [09:26:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.09, 3.00, 3.40 [09:27:54] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [09:27:54] [url] Page not found · GitHub · GitHub | github.com [09:28:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.18, 3.40 [09:28:34] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [09:28:35] [url] Page not found · GitHub · GitHub | github.com [09:29:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.09, 10.35, 10.06 [09:32:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.09, 4.37, 3.81 [09:32:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.69, 9.82, 10.03 [09:33:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.30, 11.82, 10.74 [09:33:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.86, 4.19, 3.69 [09:34:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.72, 3.71, 3.64 [09:35:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.84, 11.52, 10.77 [09:35:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.07, 7.22, 7.92 [09:36:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.13, 10.65, 10.32 [09:40:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:40:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.16, 4.20, 3.83 [09:41:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.90, 3.56 [09:42:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.97, 9.95, 10.14 [09:42:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.30, 9.83, 9.25 [09:43:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.90, 7.71, 7.80 [09:43:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.82, 3.56 [09:44:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.50, 3.80, 3.80 [09:45:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.15, 7.53, 7.72 [09:45:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.83, 3.82 [09:46:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.22, 10.00, 9.49 [09:48:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.46, 9.04, 9.93 [09:50:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.74, 9.37, 8.33 [09:51:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.43, 8.10, 7.81 [09:52:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.68, 9.08, 8.35 [09:52:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.83, 2.84, 3.31 [09:53:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.98, 7.25, 7.54 [09:53:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 2.88, 3.39 [09:55:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 2.96, 3.26 [09:56:03] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [10:01:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.28, 7.38, 7.43 [10:05:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.76, 7.17, 7.37 [10:06:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.92, 10.99, 9.96 [10:08:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.45, 10.22, 9.81 [10:09:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.10, 7.18, 7.30 [10:09:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.41, 3.31 [10:10:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.85, 9.85, 9.73 [10:11:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 2.86, 3.11 [10:12:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.99, 10.60, 10.17 [10:15:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.51, 7.63, 7.50 [10:16:42] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.05, 9.60, 9.90 [10:19:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.41, 3.71, 3.31 [10:23:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.40, 9.91, 9.00 [10:23:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.75, 3.45 [10:24:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.44, 11.03, 10.06 [10:25:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.26, 9.50, 8.98 [10:25:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.24, 2.89, 3.18 [10:26:03] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [10:26:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.23, 10.50, 9.98 [10:28:50] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [10:28:51] [url] Page not found · GitHub · GitHub | github.com [10:29:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.62, 7.93, 7.56 [10:32:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.74, 10.92, 10.29 [10:33:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.71, 7.92, 7.68 [10:34:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.05, 10.70, 10.21 [10:35:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 7.89, 8.06, 7.77 [10:35:33] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [10:35:34] [url] Page not found · GitHub · GitHub | github.com [10:36:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.73, 11.01, 10.39 [10:36:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.34, 9.41, 9.81 [10:40:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.09, 10.26, 10.08 [10:40:45] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.90, 2.69, 1.89 [10:42:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.56, 9.72, 9.07 [10:44:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.77, 9.05, 8.92 [10:44:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.33, 9.52, 9.85 [10:44:45] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.96, 1.80, 1.71 [10:46:45] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.78, 1.42, 1.58 [10:50:51] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.33, 1.74, 1.40 [10:52:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.00, 10.20, 9.97 [10:52:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.45, 1.26, 1.26 [10:53:19] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:54:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.41, 9.83, 9.86 [10:58:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 11.39, 10.96 [10:58:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.86, 3.21 [10:58:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 3.87, 3.09 [11:00:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.16, 10.56, 10.69 [11:00:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.85, 10.88, 10.23 [11:00:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.41, 3.26, 3.07 [11:00:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.03, 3.09, 2.89 [11:02:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.83, 10.96, 10.35 [11:06:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.79, 10.01, 10.12 [11:11:20] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:11:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.55, 9.97, 9.18 [11:13:53] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.35, 9.38, 9.06 [11:15:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.80, 7.48, 7.93 [11:15:33] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [11:19:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.23, 9.90, 9.84 [11:19:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.61, 8.04, 8.03 [11:20:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.77, 3.37 [11:21:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.84, 9.44, 9.68 [11:21:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.60, 7.90, 7.98 [11:21:18] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [11:23:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 7.80, 8.04, 8.03 [11:24:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.26, 3.95, 3.31 [11:25:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.92, 7.44, 7.80 [11:26:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 3.09, 3.20 [11:26:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.76, 3.31 [11:28:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.33, 3.21 [11:30:33] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [11:32:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.82, 9.31, 9.92 [11:35:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.24, 7.88, 7.80 [11:35:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.59, 10.34, 9.79 [11:36:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.61, 10.49, 10.28 [11:37:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.79, 7.43, 7.64 [11:37:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.87, 10.14, 9.78 [11:38:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.45, 9.66, 10.01 [11:40:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:41:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.14, 10.99, 10.20 [11:50:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.22, 10.53, 10.09 [11:51:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.33, 7.55, 7.38 [11:51:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.54, 10.17, 9.17 [11:51:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.83, 9.64, 10.00 [11:52:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.60, 11.18, 10.38 [11:53:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.43, 9.62, 9.09 [11:53:12] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.04, 3.10, 1.88 [11:55:12] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.71, 3.31, 2.11 [11:55:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.94, 3.32 [11:56:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.82, 11.36, 10.64 [11:57:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.69, 3.31 [11:57:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.44, 3.21 [11:59:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.22, 7.61, 7.49 [11:59:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.28, 3.35, 3.23 [11:59:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.42, 3.18, 3.15 [12:02:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [12:02:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.00, 10.27, 9.73 [12:04:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.69, 11.12 [12:04:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.30, 10.03, 9.71 [12:07:12] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.05, 3.88, 3.08 [12:09:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.54, 7.95, 7.57 [12:09:21] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:13:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.59, 10.01, 9.38 [12:15:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.80, 9.82, 9.41 [12:16:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.76, 4.32, 3.57 [12:23:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.71, 3.21, 3.45 [12:25:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.93, 3.09, 3.38 [12:36:51] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.20, 1.57, 1.24 [12:38:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.38, 1.44, 1.23 [12:40:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.57, 11.39, 11.95 [12:41:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.33, 2.89 [12:41:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.73, 3.45, 3.19 [12:42:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.51, 11.86, 12.06 [12:42:38] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [12:42:38] [url] Page not found · GitHub · GitHub | github.com [12:43:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.26, 2.84, 2.76 [12:43:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.26, 3.13, 3.12 [12:44:21] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [12:44:21] ... [12:45:12] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.64, 3.34, 3.90 [12:46:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.26, 11.34, 11.83 [12:47:12] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.20, 3.60, 3.92 [12:53:06] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [12:53:06] [url] Page not found · GitHub · GitHub | github.com [12:54:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 12.13, 11.96 [12:54:38] [02puppet] 07RhinosF1 closed pull request 03#2662: mw-sre: support py 3.11 - 13https://github.com/miraheze/puppet/pull/2662 [12:54:39] ... [12:59:50] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [12:59:50] [url] Page not found · GitHub · GitHub | github.com [12:59:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.67, 10.99, 9.90 [13:01:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.24, 9.69, 8.68 [13:03:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.38, 9.22, 8.66 [13:03:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.33, 11.73, 10.44 [13:04:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.44, 11.78, 11.97 [13:04:43] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [13:04:44] [url] Page not found · GitHub · GitHub | github.com [13:05:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 3.97, 3.49 [13:05:15] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:27] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.25, 10.29, 10.09 [13:06:06] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:11] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [13:06:12] ... [13:06:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.50, 12.48, 12.21 [13:06:29] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:42] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.48, 3.36 [13:07:35] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.55, 9.85, 9.95 [13:08:05] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 2.076 second response time [13:08:25] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.740 second response time [13:08:41] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.704 second response time [13:08:47] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mw-python-tools: major improvements - 13https://github.com/miraheze/puppet/pull/2456 [13:08:47] [url] Page not found · GitHub · GitHub | github.com [13:08:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.53, 3.38 [13:09:23] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.243 second response time [13:09:28] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.425 second response time [13:10:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.29, 3.35, 3.33 [13:11:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.21, 3.06 [13:14:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.81, 3.49 [13:16:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.62, 11.30, 9.97 [13:17:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 3.94, 3.41 [13:18:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.45, 10.29, 9.77 [13:18:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.62, 3.48 [13:19:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.76, 3.40 [13:19:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.73, 10.20, 9.37 [13:20:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.27, 9.61, 9.58 [13:21:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.99, 9.75, 9.31 [13:23:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.59, 10.24, 9.75 [13:24:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.74, 9.66, 9.60 [13:25:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.85, 3.67 [13:27:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.92, 3.68 [13:31:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.35, 10.82, 9.82 [13:33:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.20, 4.36, 3.91 [13:35:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.26, 3.58, 3.68 [13:35:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.92, 11.90, 10.52 [13:36:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.98, 3.70 [13:39:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.70, 3.68 [13:39:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.94, 12.13, 10.91 [13:40:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.79, 3.71 [13:41:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.45, 3.58 [13:41:20] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:41:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.42, 3.46 [13:44:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.62, 4.41, 3.94 [13:45:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.51, 4.20, 3.83 [13:45:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.41, 11.32, 10.93 [13:46:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.73, 3.86, 3.74 [13:47:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.85, 7.34, 7.96 [13:47:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 12.00, 10.83, 9.66 [13:49:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.19, 10.12, 9.53 [13:49:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.60, 3.63 [13:50:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.81, 3.86 [13:51:07] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.18, 7.85, 7.99 [13:53:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.38, 7.38, 7.81 [13:53:08] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 3.852 second response time [13:54:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.46, 2.94, 3.37 [13:55:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 3.35, 3.43 [13:57:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.31, 1.84, 1.47 [13:57:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.42, 7.53, 7.73 [13:58:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.78, 1.46, 1.38 [13:59:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.68, 3.56 [14:00:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.92, 2.94, 3.37 [14:00:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 4.10, 3.74 [14:01:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.47, 10.83, 10.62 [14:03:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.04, 7.65, 7.76 [14:03:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.12, 10.56, 10.58 [14:04:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.48, 3.49 [14:04:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.65, 3.63 [14:06:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.43, 3.46 [14:07:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.92, 9.03, 9.93 [14:08:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 3.03, 3.31 [14:08:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.38, 3.49 [14:09:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 4.00, 3.66 [14:10:02] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:40] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:47] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:46] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:56] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.33, 5.25, 4.16 [14:13:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.20, 7.16, 7.40 [14:13:49] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.693 second response time [14:14:01] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.778 second response time [14:14:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.52, 10.74, 11.83 [14:14:52] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.406 second response time [14:14:58] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.135 second response time [14:15:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.54, 7.09, 7.33 [14:16:17] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27066 bytes in 2.560 second response time [14:17:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.27, 7.78, 7.57 [14:21:12] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.03, 3.06, 3.96 [14:21:23] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:12] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:16] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:42] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:16] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.17, 11.37, 11.47 [14:26:25] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.343 second response time [14:26:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.45, 9.69, 9.21 [14:26:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.71, 3.99 [14:28:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.94, 11.86, 11.66 [14:28:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.72, 8.82, 8.95 [14:28:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.50, 4.05, 4.10 [14:29:03] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 8.960 second response time [14:29:30] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.316 second response time [14:29:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.24, 10.41, 9.77 [14:29:45] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.459 second response time [14:30:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.03, 12.15, 11.78 [14:30:25] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.641 second response time [14:30:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.45, 3.39, 3.85 [14:31:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.33, 11.53, 10.27 [14:32:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 11.93, 11.75 [14:32:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.77, 4.21, 4.08 [14:33:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.69, 11.46, 10.39 [14:34:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.92, 12.08, 11.81 [14:34:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.58, 3.57, 3.86 [14:35:12] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 3.17, 3.10, 3.37 [14:37:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.89, 10.06, 10.00 [14:38:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.51, 3.99 [14:40:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.19, 4.47, 4.28 [14:40:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 11.30, 11.72 [14:40:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.05, 4.15, 3.93 [14:41:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.23, 10.20, 10.08 [14:42:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.42, 10.27, 9.58 [14:48:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.90, 1.33 [14:49:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.77, 1.47, 1.24 [14:50:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.34, 9.94, 9.78 [14:51:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.50, 4.00 [14:52:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 11.42, 11.39 [14:53:34] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:48] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:03] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.84, 11.53, 11.43 [14:54:36] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:01] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.85, 8.88, 9.84 [14:57:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.71, 5.06, 4.35 [14:58:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.77, 2.04, 1.61 [14:59:27] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:30] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:13] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:13] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:20] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:24] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:27] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.52, 1.86, 1.59 [15:00:52] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:11] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:24] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.568 second response time [15:01:29] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.263 second response time [15:02:07] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.402 second response time [15:02:16] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 8.553 second response time [15:02:24] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.06, 1.55, 1.51 [15:02:28] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.378 second response time [15:03:00] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.077 second response time [15:03:00] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.798 second response time [15:03:10] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.147 second response time [15:03:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.38, 10.24, 10.04 [15:04:15] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.451 second response time [15:04:24] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.569 second response time [15:04:27] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.197 second response time [15:05:20] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 1.408 second response time [15:06:38] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:26] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:32] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.79, 9.58, 9.84 [15:09:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.51, 7.46, 7.95 [15:11:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.08, 8.16, 8.15 [15:12:15] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.894 second response time [15:13:08] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.029 second response time [15:13:40] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.633 second response time [15:13:40] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.313 second response time [15:14:58] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 0.509 second response time [15:17:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.13, 10.52, 10.00 [15:19:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.43, 10.21, 9.96 [15:22:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 11.14, 10.70 [15:24:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.79, 11.05, 10.73 [15:25:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.53, 10.11, 10.03 [15:33:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.78, 11.15, 10.47 [15:35:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.45, 7.09, 7.76 [15:36:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.80, 11.45, 10.85 [15:37:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.08, 7.36, 7.76 [15:37:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.32, 10.06, 10.16 [15:39:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.66, 7.45, 7.75 [15:40:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.59, 10.38, 10.60 [15:41:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.26, 7.63, 7.77 [15:42:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.81, 10.70, 9.96 [15:44:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.37, 9.94, 9.78 [15:47:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.93, 7.46, 7.67 [15:48:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.20, 10.39, 9.97 [15:50:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.09, 11.18, 10.32 [15:51:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.37, 7.78, 7.73 [15:51:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.69, 10.02, 9.84 [15:53:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.76, 7.59, 7.68 [15:53:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.68, 9.88, 9.82 [15:56:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.43, 11.48, 10.89 [15:56:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.69, 10.86, 10.57 [15:56:55] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:01] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.46, 8.38, 8.00 [15:57:16] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:23] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:28] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.96, 1.53, 1.28 [15:57:34] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 10.41, 10.57 [15:58:59] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.571 second response time [15:59:06] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.847 second response time [15:59:23] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 7.156 second response time [15:59:24] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.489 second response time [15:59:25] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.02, 1.31, 1.23 [15:59:34] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.213 second response time [16:02:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [16:02:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.80, 10.22, 9.93 [16:04:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 11.28, 10.87 [16:04:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.46, 11.17, 10.68 [16:06:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.35, 10.01, 9.98 [16:06:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.30, 11.05, 10.69 [16:08:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.58, 11.13, 10.94 [16:09:20] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [16:12:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.83, 11.51, 10.94 [16:14:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.15, 10.80, 10.75 [16:22:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.73, 9.10, 9.96 [16:32:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.07, 10.69 [16:34:17] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:38:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.16, 11.40, 11.05 [16:39:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.23, 10.27, 10.02 [16:43:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.27, 11.45, 10.57 [16:45:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.70, 10.94, 10.50 [16:50:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.39, 9.77, 8.83 [16:51:17] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.66, 9.52, 10.02 [16:52:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.49, 9.46, 8.83 [16:53:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.34, 3.81, 2.20 [16:54:33] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:37] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.69, 10.23, 10.19 [16:56:37] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.467 second response time [16:56:44] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.754 second response time [16:57:13] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.56, 10.04, 10.13 [16:58:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.69, 1.79, 1.79 [17:00:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.80, 1.48, 1.67 [17:02:19] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [17:02:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.54, 10.33, 10.03 [17:04:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.70, 8.66, 9.96 [17:04:34] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.94, 5.62, 3.26 [17:04:57] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.05, 9.18, 9.62 [17:04:59] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.39, 6.56, 7.64 [17:05:14] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:19] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:26] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:17] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:34] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.71, 5.83, 3.62 [17:08:15] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.177 second response time [17:09:26] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.056 second response time [17:09:32] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.978 second response time [17:09:39] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 8.336 second response time [17:11:14] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.992 second response time [17:15:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.23, 7.68, 7.59 [17:18:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.89, 10.29, 10.12 [17:18:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.83, 10.57, 9.80 [17:20:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.28, 9.38, 8.40 [17:20:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.50, 11.01, 9.88 [17:31:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.16, 9.85, 8.71 [17:31:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.47, 11.09, 10.04 [17:31:17] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.15, 11.03, 10.20 [17:31:17] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.23, 12.21, 10.57 [17:31:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 11.25, 10.59 [17:31:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.24, 10.81, 10.36 [17:31:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.83, 11.24, 10.69 [17:31:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.45, 11.34, 10.85 [17:34:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 11.15, 10.71 [17:38:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.39, 11.31, 10.90 [17:38:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.99, 3.51, 2.03 [17:40:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.49, 11.22, 10.93 [17:41:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.57, 7.28, 7.82 [17:42:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.52, 1.95, 1.73 [17:42:38] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.19, 9.02, 9.95 [17:43:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.04, 7.94, 7.99 [17:44:24] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.59, 1.62 [17:46:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.91, 11.79, 11.29 [17:48:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 10.91, 10.81 [17:50:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.62, 10.98, 10.84 [17:50:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.89, 9.83, 9.05 [17:52:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.27, 11.74, 11.13 [17:52:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.37, 9.58, 9.05 [17:54:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.13, 11.48, 11.11 [17:58:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.08, 11.63, 11.24 [18:01:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.91, 10.45, 10.02 [18:02:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.10, 11.22, 11.21 [18:03:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.54, 9.90, 9.89 [18:09:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.91, 1.38 [18:10:39] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.84, 1.57, 1.32 [18:11:48] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:12] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:58] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:59] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.07, 7.60, 7.97 [18:14:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.15, 9.44, 10.14 [18:14:47] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.073 second response time [18:15:00] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.306 second response time [18:15:01] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.829 second response time [18:15:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.40, 7.87, 8.02 [18:15:51] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [18:16:20] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 2.644 second response time [18:17:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.09, 7.60, 7.91 [18:21:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.83, 7.99, 7.97 [18:22:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.27, 12.07, 11.59 [18:23:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.72, 7.83, 7.91 [18:24:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.18, 11.69, 11.50 [18:25:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.24, 7.76, 7.87 [18:29:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.67, 7.59, 7.81 [18:29:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.49, 9.84, 8.99 [18:31:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.98, 9.05, 8.80 [18:32:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.49, 11.51 [18:32:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.12, 10.49, 10.03 [18:33:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.94, 10.22, 9.43 [18:34:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.98, 11.29, 11.44 [18:35:10] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.00, 9.73, 9.35 [18:36:15] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.20, 9.40, 9.71 [18:36:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.02, 11.85, 11.62 [18:37:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.61, 7.75, 7.74 [18:39:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.99, 7.06, 7.49 [18:40:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.96, 10.87, 10.25 [18:40:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 11.40, 11.51 [18:40:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.14, 9.32, 8.11 [18:40:56] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:56] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:26] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:40] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:57] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:02] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.50, 11.50, 10.54 [18:42:19] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:47] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.81, 8.32, 7.89 [18:43:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.64, 7.26, 7.42 [18:43:39] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.338 second response time [18:43:59] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.708 second response time [18:44:04] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 4.717 second response time [18:44:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.70, 11.23, 10.58 [18:44:15] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.333 second response time [18:44:47] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.196 second response time [18:44:55] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [18:44:58] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.483 second response time [18:45:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.26, 7.20, 7.37 [18:45:31] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.344 second response time [18:50:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.69, 11.60, 11.44 [18:54:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.87, 9.55, 10.12 [18:54:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.90, 11.82, 11.58 [18:54:51] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.12, 1.44, 1.00 [18:55:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.00, 7.85, 7.48 [18:56:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.19, 1.37, 1.03 [18:57:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.67, 7.32, 7.33 [18:58:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.20, 12.23, 11.76 [19:00:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.87, 11.70, 11.63 [19:02:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.57, 10.01, 9.43 [19:03:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 2.08, 1.49 [19:04:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.12, 9.47, 9.32 [19:06:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.42, 12.42, 11.83 [19:08:36] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [19:09:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.29, 8.16, 7.49 [19:09:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.83, 1.58 [19:10:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 11.29, 11.50 [19:11:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.23, 1.57, 1.50 [19:11:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.99, 10.44, 10.05 [19:13:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.58, 11.56, 10.53 [19:15:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.85, 11.55, 10.64 [19:18:52] RhinosF1, Argument 1 passed to MediaWiki\User\UserGroupManager::getUserGroups() must implement interface MediaWiki\User\UserIdentity, instance of MediaWiki\Extension\CentralAuth\User\CentralAuthGroupMembershipProxy given, called in /srv/mediawiki/w/extensions/DiscordNotifications/includes/Hooks.php on line 476 [19:19:23] Voidwalker: ah ye! [19:19:30] I'll check upstream later [19:21:01] Look's like we're using a fork from Universal-Omega that should have a partial fix [19:21:13] [discord] Yeah, it seems like the log entry failed but the change went through [19:21:26] Log is always the last thing [19:21:48] Voidwalker: I think iirc it's over strict typing in core [19:21:59] Not a DiscordNotifications issue [19:22:05] Reception123: wanna add yourself and agent to stewards@? [19:22:13] getUserGroups() can't work with global rights [19:22:16] MacFan4000: lol you said that exactly as I did [19:22:21] !log added myself and agent to stewards@ [19:22:25] the timing was so precise :P [19:22:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.28, 10.09, 9.69 [19:22:54] !log added agent to cvt@ [19:22:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:52] Voidwalker: we should do flags in the cvt channels [19:24:20] Voidwalker: https://phabricator.wikimedia.org/T302408 should have the history [19:24:21] [url] ⚓ T302408 Argument 1 passed to HAWelcomeHooks::onUserGroupsChanged() must be an instance of User, instance of CentralAuthGroupMembershipProxy given, called in /srv/mediawiki/w/includes/HookContainer/H | phabricator.wikimedia.org [19:24:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.59, 9.39, 9.48 [19:25:27] Not entirely the same actually [19:25:45] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.11, 11.12, 10.87 [19:25:57] This is an issue with strict typing for UserGroupManager::getUserGroups, as opposed to the hook [19:26:58] Voidwalker: ye [19:27:07] I think we need to get them to update the type [19:27:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.82, 10.42, 10.66 [19:27:52] or have the extension handle it differently [19:28:06] *DiscordNotifications [19:33:09] Voidwalker: I asked taavi [19:34:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.79, 11.46, 11.18 [19:35:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.95, 9.31, 10.07 [19:36:20] Voidwalker: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/811424 [19:36:34] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:37:37] ah, I was going to suggest we simply remove the userGroupManager and swap into using the $newUGMs variable instead [19:38:02] Voidwalker: +2'd [19:38:13] You'll have to update when jerkins merges [19:39:13] that'll be easier :) will do [19:40:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.13, 11.35, 11.31 [19:41:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.77, 10.38, 10.20 [19:42:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.18, 3.89 [19:43:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.64, 6.97, 7.72 [19:43:54] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/ed2a9b93a63c...6e353a035b94 [19:43:54] [url] Comparing ed2a9b93a63c...6e353a035b94 · miraheze/dns · GitHub | github.com [19:43:55] [02miraheze/dns] 07Reception123 036e353a0 - add drowiki.com zone [19:43:58] Voidwalker: and merged [19:44:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.25, 4.61, 4.35 [19:44:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.49, 11.35 [19:44:38] all right, let me see if I can update :P [19:45:01] Cool [19:45:09] Let me know if you need help with deploy tool [19:45:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.95, 9.53, 9.93 [19:46:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 10.77, 11.10 [19:49:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.86, 7.86, 7.81 [19:49:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.23, 10.27, 9.02 [19:50:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 10.93, 11.05 [19:51:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.88, 11.10, 9.49 [19:51:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.73, 10.79, 10.30 [19:51:47] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.69, 1.79, 1.46 [19:52:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.41, 10.82, 10.98 [19:53:41] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.79, 1.44, 1.37 [19:53:51] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/6e353a035b94...2b929676f858 [19:53:51] [url] Comparing 6e353a035b94...2b929676f858 · miraheze/dns · GitHub | github.com [19:53:52] [02miraheze/dns] 07Reception123 032b92967 - add dkpedia.com zone [19:54:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 11.43, 11.18 [19:57:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.04, 9.81, 9.43 [19:57:28] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://github.com/miraheze/ssl/compare/498904b976ef...98e3c219064f [19:57:28] [url] Comparing 498904b976ef...98e3c219064f · miraheze/ssl · GitHub | github.com [19:57:29] [02miraheze/ssl] 07Reception123 03224cbe6 - add slupe.fp64.org [19:57:31] [02miraheze/ssl] 07Reception123 03517dcb4 - add dkpedia.com cert [19:57:32] [02miraheze/ssl] 07Reception123 0398e3c21 - add drowiki.com cert [19:59:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.95, 7.35, 7.76 [20:00:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.46, 11.48, 11.37 [20:00:26] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/98e3c219064f...e86320449e70 [20:00:27] [url] Comparing 98e3c219064f...e86320449e70 · miraheze/ssl · GitHub | github.com [20:00:28] [02miraheze/ssl] 07Reception123 03e863204 - donkeykongwiki force redirect [20:01:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.41, 11.35, 10.14 [20:01:47] my local git is completely broken, can someone else handle? [20:02:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [20:02:22] Voidwalker: what needs doing? [20:02:53] we have an update for CentralAuth that should fix fatals on changing global group membership [20:02:55] Reception123: bump CA [20:03:22] ok, can do [20:03:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.79, 11.56, 10.80 [20:03:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.41, 10.48, 9.97 [20:07:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.97, 11.57, 10.96 [20:08:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 12.10, 11.65 [20:09:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.90, 10.50, 9.88 [20:10:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 11.83, 11.61 [20:11:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.91, 9.88, 9.72 [20:11:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.86, 9.45, 9.79 [20:13:54] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.55, 7.20, 7.22 [20:15:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.67, 7.07, 7.18 [20:19:47] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.20, 6.00, 6.71 [20:22:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 11.04, 11.16 [20:22:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.02, 10.92, 10.85 [20:23:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.93, 6.55, 6.82 [20:24:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.71, 10.55, 10.95 [20:24:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.24, 10.65, 10.76 [20:25:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.20, 6.56, 6.80 [20:28:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.73, 11.59, 11.29 [20:32:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.39, 11.03, 11.17 [20:33:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.65, 1.75, 1.46 [20:35:37] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.31, 7.30, 6.90 [20:36:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.43, 1.39 [20:37:07] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.15, 9.76, 10.14 [20:37:32] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.04, 7.21, 6.92 [20:39:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.38, 3.10, 3.94 [20:41:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.57, 4.01 [20:43:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.80, 10.81, 10.44 [20:45:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.20, 9.66, 10.07 [20:45:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.52, 7.43, 7.10 [20:47:21] help! [20:47:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.49, 7.52, 7.18 [20:47:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.59, 10.95, 10.83 [20:51:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.03, 8.40, 7.58 [20:51:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.46, 11.31, 11.02 [20:52:09] ping Voidwalker Reception123 [20:53:03] ping Agent [20:53:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.35, 7.42, 7.34 [20:54:47] Hispano76: what's up? [20:55:28] Agent, I have not been granted the bureaucrat flag. https://ucronias.miraheze.org/wiki/Especial:ListaUsuarios [20:55:29] [url] Lista de usuarios - Ucronías | ucronias.miraheze.org [20:55:38] Let me see [20:55:45] [[Special:UserRights]] [20:55:45] https://meta.miraheze.org/wiki/Special:UserRights [20:56:53] [discord] Are doubled notifications about "first edit" known issue? [20:57:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [20:59:07] Hispano76: I can't seem to find in what log entry you created the wiki [20:59:57] system bug... Agent RhinosF1 [21:00:06] I see [21:00:27] I'll grant the rights I guess since you and Rhinos seem to be the only users registered on there [21:01:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.39, 7.85, 7.49 [21:02:25] For future reference Hispano76, I would advise you use RequestWiki to request the wiki and to leave a paper trail [21:02:27] Anyway, done [21:02:51] thanks xd [21:05:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.49, 7.61, 7.55 [21:12:18] Thanks Agent [21:15:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.51, 7.75, 7.56 [21:17:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.65, 3.96 [21:17:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.51, 7.65, 7.55 [21:19:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.02, 8.14, 7.74 [21:21:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.98, 4.01 [21:23:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.99, 3.43, 3.82 [21:25:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 4.20, 4.05 [21:25:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.61, 7.18, 7.48 [21:28:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.33, 3.38, 3.83 [21:29:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.33, 3.77 [21:31:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.82, 3.89 [21:33:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.76, 9.29, 10.08 [21:34:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.93, 3.91, 3.85 [21:35:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.94, 4.00 [21:37:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 4.19, 4.08 [21:37:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.38, 9.97, 10.25 [21:37:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.50, 7.72, 7.37 [21:38:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.90, 3.87 [21:39:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.64, 9.58, 10.05 [21:39:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.61, 3.84 [21:39:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.07, 7.44, 7.31 [21:40:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.99, 3.90 [21:40:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.21, 10.56, 9.74 [21:41:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.09, 4.18, 4.02 [21:41:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.54, 8.20, 7.60 [21:43:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.56, 10.37, 10.29 [21:43:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.33, 7.54, 7.44 [21:45:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 9.76, 10.07 [21:46:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.03, 9.68, 9.70 [21:51:25] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.55, 6.13, 6.76 [21:53:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 10.23, 10.14 [21:54:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.51, 10.67, 10.05 [21:55:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.49, 9.52, 9.88 [21:58:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.74, 1.31 [21:59:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 10.72, 10.25 [22:00:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.54, 1.29 [22:01:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.58, 10.18, 10.11 [22:02:47] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.76, 9.96, 10.01 [22:13:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.48, 3.94 [22:14:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.20, 3.30, 3.86 [22:17:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.43, 4.33, 4.11 [22:18:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.10, 5.10, 4.40 [22:21:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.81, 6.93, 6.63 [22:23:10] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.83, 1.53 [22:23:25] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.76, 6.52, 6.52 [22:25:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.19, 1.64, 1.49 [22:29:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.49, 3.90 [22:30:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.29, 3.47, 4.00 [22:33:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.30, 3.78 [22:34:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.43, 3.84 [22:35:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 10.84, 10.03 [22:36:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.46, 3.80 [22:37:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.41, 10.31, 9.92 [22:38:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.49, 4.21, 4.03 [22:39:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.41, 10.02, 9.87 [22:39:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.31, 3.53 [22:40:10] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.53, 7.17, 6.71 [22:40:49] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:41:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.31, 3.50 [22:42:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.63, 3.82 [22:43:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.68, 4.14, 3.77 [22:45:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.59, 3.61 [22:47:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.46, 10.06, 9.86 [22:49:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.55, 3.51, 3.50 [22:49:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.51, 3.56 [22:51:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.41, 3.45 [22:51:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.86, 9.87, 9.88 [22:51:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 12.42, 9.02, 7.68 [22:53:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.95, 3.19, 3.37 [22:53:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.09, 3.39 [22:54:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.04, 3.34 [22:56:16] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.65, 3.40, 1.87 [22:57:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.30, 3.31, 3.43 [23:00:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.41, 1.78, 1.55 [23:01:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.72, 4.26, 3.75 [23:01:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.37, 7.87, 7.76 [23:02:10] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.16, 1.25, 1.38 [23:02:57] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 7.54, 5.46, 3.85 [23:05:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.53, 3.36, 3.50 [23:06:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.87, 11.52, 10.54 [23:07:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.54, 3.93, 3.68 [23:07:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.13, 8.27, 7.93 [23:08:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 3.89, 3.54 [23:08:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.68, 3.43 [23:09:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.63, 3.60 [23:09:34] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [23:09:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 10.49, 10.37 [23:10:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.73, 3.53 [23:10:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.87, 1.59 [23:10:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.29, 3.31 [23:11:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.39, 7.34, 7.64 [23:12:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.67, 4.55, 3.85 [23:12:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.55, 1.51 [23:14:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.75, 3.49 [23:16:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.80, 3.71 [23:18:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.49, 4.44, 3.93 [23:20:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.80, 3.66 [23:22:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.89, 3.82 [23:22:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.18, 5.80, 5.57 [23:23:25] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.11, 8.06, 7.63 [23:23:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.04, 11.84, 11.03 [23:24:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 4.01, 3.87 [23:25:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.97, 11.47, 10.98 [23:26:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.60, 3.73 [23:27:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.74, 3.63 [23:27:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.63, 7.81, 7.67 [23:28:57] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.58, 5.95, 5.66 [23:29:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 11.58, 11.12 [23:31:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.53, 3.59 [23:31:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.05, 11.45, 11.14 [23:32:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 3.75, 3.73 [23:32:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.96, 3.73 [23:32:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.46, 5.94, 5.72 [23:34:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.72, 3.72 [23:35:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 11.59, 11.24 [23:36:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.74, 3.71 [23:36:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.45, 3.60 [23:37:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.35, 11.83, 11.39 [23:38:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.65, 3.32, 3.56 [23:38:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.93, 2.94, 3.39 [23:39:25] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.22, 6.19, 6.72 [23:42:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.85, 2.68, 3.25 [23:43:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 2.92, 3.30 [23:47:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.55, 3.62, 3.49 [23:49:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 3.32, 3.40 [23:49:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.38, 3.34 [23:50:57] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.04, 5.73, 5.67 [23:51:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 2.99, 3.19 [23:52:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.78, 5.57, 5.60 [23:53:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.44, 9.45, 10.12 [23:55:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.24, 3.29 [23:57:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 3.83, 3.49 [23:58:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.57, 3.40 [23:58:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.57, 3.37 [23:59:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.49, 3.40