[00:01:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.14, 2.34, 2.10 [00:01:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.98, 3.67 [00:03:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.14, 4.83, 4.02 [00:04:21] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.62, 3.24, 3.04 [00:06:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.01, 3.08, 3.00 [00:09:44] ping CosmicAlpha [00:11:33] Hispano76: replied in PM. [00:13:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.25, 9.06, 8.36 [00:14:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.17, 8.45, 8.21 [00:23:01] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dnd.bellinrattin.it could not be found [00:33:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.15, 3.44, 3.95 [00:37:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.80, 3.94 [00:51:12] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.14, 2.22, 1.50 [00:53:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.61, 1.99, 1.50 [00:55:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.94, 1.60, 1.41 [00:58:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.30, 10.23, 9.41 [01:01:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.41, 2.16, 1.63 [01:03:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.77, 1.66, 1.51 [01:04:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.15, 10.85, 9.93 [01:06:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.73, 10.87, 10.06 [01:07:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.23, 10.21, 9.08 [01:15:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.82, 10.07, 9.54 [01:15:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.72, 9.37, 9.85 [01:21:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.58, 10.23, 10.04 [01:23:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 10.99, 10.35 [01:27:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.48, 11.25, 10.59 [01:32:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.21, 9.96, 9.01 [01:33:14] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.64, 3.52, 3.10 [01:34:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.16, 9.41, 8.93 [01:35:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.05, 11.49, 10.94 [01:37:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.17, 11.07, 10.87 [01:39:05] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.46, 3.15, 3.10 [01:44:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.64, 10.26, 9.51 [01:44:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.01, 10.18, 9.54 [01:46:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.85, 9.53, 9.35 [01:46:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.06, 9.53, 9.39 [01:52:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.18, 2.24, 1.61 [01:53:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.54, 3.95 [01:54:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.06, 1.74, 1.50 [01:55:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.87, 9.64, 10.14 [01:55:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.84, 4.00 [01:56:34] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.05, 1.55, 1.45 [01:58:29] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.42, 3.25, 3.14 [02:00:26] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.80, 3.09, 3.09 [02:41:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.69, 10.28, 9.62 [02:47:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.21, 10.14, 9.84 [02:47:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.70, 3.99 [02:57:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.37, 10.65, 10.22 [02:57:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.76, 3.79 [02:59:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.52, 3.70 [03:01:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.41, 10.11, 10.10 [03:05:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.82, 10.73, 10.37 [03:07:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.63, 11.81, 10.81 [03:09:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.41, 11.52, 10.83 [03:13:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.14, 3.81, 3.57 [03:14:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.86, 1.46 [03:15:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 3.22, 3.38 [03:16:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.55, 1.39 [03:16:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 3.32, 3.91 [03:17:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.10, 9.57, 10.13 [03:20:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.88, 3.78, 3.91 [03:21:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.61, 3.99 [03:22:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.39, 3.22, 3.69 [03:24:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.64, 3.79 [03:25:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.57, 3.85 [03:26:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.30, 3.65 [03:27:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.07, 3.62 [03:28:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.36, 3.61 [03:30:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.12, 3.07, 3.49 [03:31:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.51, 2.61, 3.35 [03:32:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 2.68, 3.28 [03:42:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.50, 3.35 [03:43:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.53, 3.40 [03:44:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 4.00, 3.55 [03:45:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.56, 3.23 [03:46:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.70, 3.48 [03:48:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 4.19, 3.71 [03:49:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 4.18, 3.72 [03:49:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.96, 4.14, 3.55 [03:52:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.86, 3.69 [03:55:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.49, 3.57 [03:56:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 3.80, 3.66 [03:57:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.95, 3.97, 3.73 [03:58:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.85, 3.71 [04:00:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.82, 3.71 [04:01:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.78, 3.70 [04:03:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 4.02, 3.79 [04:04:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.26, 3.08, 3.40 [04:04:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.54, 9.68, 8.55 [04:04:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.18, 2.96, 3.39 [04:05:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.49, 3.62 [04:06:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.26, 8.79, 8.37 [04:12:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.99, 10.05, 9.01 [04:12:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.75, 9.33, 8.20 [04:14:59] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.93, 9.48, 8.39 [04:15:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.70, 3.82, 3.61 [04:15:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.61, 10.05, 9.35 [04:16:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.69, 9.46, 9.02 [04:17:24] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.16, 9.46, 9.20 [04:19:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.64, 3.61 [04:21:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.87, 3.70 [04:23:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.91, 3.76 [04:24:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.44, 3.26 [04:26:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.14, 3.19, 3.19 [04:27:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 10.94, 10.00 [04:29:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.12, 2.79, 3.29 [04:29:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.53, 11.07, 10.16 [05:01:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.37, 8.90, 9.79 [05:05:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 10.62, 10.24 [05:07:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.32, 9.94, 10.03 [05:12:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.73, 10.57, 9.14 [05:14:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.25, 9.69, 9.00 [05:18:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 4.04, 3.36 [05:18:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.48, 2.87 [05:20:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.85, 3.12 [05:21:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.65, 1.25 [05:23:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.48, 1.24 [05:24:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 3.01, 2.95 [05:24:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.58, 3.44 [05:26:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.21, 3.38, 3.37 [05:28:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.50, 3.31, 3.14 [05:32:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 4.00, 3.70, 3.47 [05:34:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.69, 3.01, 3.25 [05:39:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.48, 2.09, 1.56 [05:40:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.43, 3.09 [05:41:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.75, 1.60, 1.44 [05:42:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.16, 3.02 [05:44:34] !log DELETED and DROPPED ucroniaswiki (reset) [05:44:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:05:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.73, 3.24 [06:07:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.38, 3.17 [06:11:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.56, 3.31 [06:11:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.66, 10.51, 9.73 [06:13:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.18, 3.20 [06:17:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.56, 3.66, 3.38 [06:17:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 11.65, 10.51 [06:19:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.19, 3.23 [06:23:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 11.29, 10.85 [06:25:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 11.47, 10.96 [06:27:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.46, 11.03, 10.86 [06:28:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.01, 3.39, 2.92 [06:29:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.39, 3.24, 2.91 [06:33:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 11.01, 10.85 [06:33:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.52, 3.13 [06:35:31] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.62, 2.69, 1.83 [06:35:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 3.25, 3.08 [06:37:31] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 3.10, 2.85, 2.00 [06:39:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.00, 11.10, 11.12 [06:41:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.91, 3.49 [06:43:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 3.22, 3.28 [06:47:13] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.20, 7.75, 6.72 [06:49:12] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.66, 7.63, 6.80 [06:53:09] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.53, 6.45, 6.49 [06:53:31] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.51, 3.51, 2.82 [06:55:31] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.66, 3.50, 2.90 [06:56:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.47, 1.86, 1.30 [06:57:31] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.51, 3.12, 2.83 [06:57:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.46, 1.22 [06:59:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.43, 11.25, 10.71 [07:01:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.14, 11.33, 10.82 [07:05:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 11.56, 10.98 [07:05:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.09, 1.86, 1.53 [07:07:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.67, 11.09, 10.90 [07:07:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.62, 1.49 [07:09:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.36, 12.15, 11.32 [07:11:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.43, 10.62, 10.85 [07:12:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.33, 10.69, 9.54 [07:14:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.34, 10.10, 9.46 [07:23:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.94, 10.77, 10.55 [07:25:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 10.37, 10.44 [07:25:43] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.56, 1.70, 1.19 [07:27:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.15, 9.62, 10.15 [07:27:38] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.73, 1.30, 1.10 [07:32:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.25, 9.60, 8.82 [07:34:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.74, 9.31, 8.81 [07:42:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.06, 9.72, 8.94 [07:44:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.19, 8.91, 8.73 [07:48:03] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:58] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.25, 10.61, 10.19 [07:49:42] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:49:57] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:00] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:19] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 26.66, 13.20, 6.39 [07:51:36] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.546 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [07:53:08] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.321 second response time [07:53:57] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.319 second response time [07:54:02] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.041 second response time [07:54:15] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 27073 bytes in 0.642 second response time [07:54:24] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.029 second response time [07:59:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.44, 11.86, 10.93 [08:01:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 11.29, 10.83 [08:05:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 11.39, 10.94 [08:06:10] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.53, 2.40, 3.97 [08:06:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.37, 3.09 [08:07:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 3.92, 3.28 [08:07:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.17, 10.77, 10.75 [08:08:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.33, 3.26, 3.08 [08:09:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.82, 3.32 [08:10:00] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.50, 1.89, 3.40 [08:10:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.97, 9.07, 8.42 [08:11:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.47, 4.15, 3.48 [08:14:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.00, 9.48, 8.79 [08:19:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.76, 3.65 [08:19:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.02, 9.33, 10.09 [08:19:40] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:19:53] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 9.98, 5.07, 3.85 [08:19:57] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:00] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:09] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:16] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.57, 3.21 [08:22:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.73, 3.29 [08:23:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.96, 10.92, 10.57 [08:23:51] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.666 second response time [08:23:57] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.324 second response time [08:24:02] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.042 second response time [08:24:13] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 27073 bytes in 0.338 second response time [08:24:20] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.029 second response time [08:24:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.70, 3.34, 3.21 [08:27:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.25, 3.04, 3.33 [08:27:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.25, 9.64, 10.16 [08:33:38] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.56, 1.83, 3.66 [08:35:33] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.26, 1.33, 3.26 [08:41:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.44, 3.14 [08:43:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.69, 3.27 [08:45:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 4.39, 3.60 [08:47:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.87, 3.50 [08:49:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 3.24, 3.32 [08:56:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.81, 3.23 [09:01:37] PROBLEM - puppet111 Current Load on puppet111 is WARNING: WARNING - load average: 5.66, 4.35, 2.95 [09:02:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.12, 3.27, 3.17 [09:03:37] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 1.91, 3.32, 2.74 [09:10:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.43, 3.09 [09:12:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 3.07, 2.99 [09:18:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.11, 3.02 [09:20:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 2.80, 2.92 [09:23:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.83, 3.32 [09:25:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.69, 3.32 [09:27:00] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.51, 1.75, 1.44 [09:27:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.60, 2.97, 3.11 [09:29:00] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.73, 1.35, 1.33 [09:40:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 10.25, 9.29 [09:42:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.99, 10.00, 9.31 [09:42:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.09, 12.02, 9.83 [09:44:29] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.21, 6.63, 5.75 [09:44:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.65, 11.71, 9.98 [09:45:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.73, 3.30 [09:46:29] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.83, 6.24, 5.71 [09:47:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 3.33, 3.22 [09:50:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.22, 10.17, 9.87 [09:53:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.84, 3.32 [09:55:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.96, 3.13, 3.12 [09:55:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.90, 10.86, 10.08 [09:58:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 3.69, 3.34 [09:58:54] RhinosF1: this is strange [09:58:54] https://phabricator.miraheze.org/T9514#192423 [09:58:55] [url] ⚓ T9514 DB error when accessing Special:Contributions on horriblemusicwiki | phabricator.miraheze.org [09:59:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 4.07, 3.51 [09:59:10] Could it be related to the actor migrations script running perhaps? [09:59:53] Reception123: unlikely because it's only writing new data [10:00:02] Amir backported some fixes though [10:00:08] hmm, that is true [10:00:22] I don't remember seeing such an issue though. Any idea what we can do? [10:00:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.44, 2.86 [10:00:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 3.21, 3.21 [10:01:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.76, 3.46 [10:01:08] https://github.com/miraheze/mediawiki/commit/3d17c9d4e881e4d6815e7abc5f858569978da98b & https://github.com/miraheze/mediawiki/commit/e50e5147da123d358c8826fb3fb87fc3b5b7d54f [10:01:08] [url] RecentChange: Straight join to actor table when needed · miraheze/mediawiki@3d17c9d · GitHub | github.com [10:01:09] [url] RecentChange: Make join to comment table also straight · miraheze/mediawiki@e50e514 · GitHub | github.com [10:01:13] Reception123: pull them ^ [10:01:24] Or just update mediawiki to the head of 1.38 [10:01:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.83, 9.97, 9.95 [10:01:37] CosmicAlpha: wanted to wait because nothing important [10:01:41] But that changes [10:01:56] That exception is a safety measure [10:02:00] So the farm doesn't go down [10:02:29] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.33, 7.30, 6.39 [10:02:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.90, 3.54 [10:04:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 3.26, 2.92 [10:04:29] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.90, 7.13, 6.45 [10:04:43] I'll be out for a bit but will get to it later hopefully, if not CosmicAlpha can do it before [10:04:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.94, 3.45, 3.41 [10:05:01] Them 2 commits are most important [10:05:10] If you don't want to pull everything [10:06:29] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.75, 6.65, 6.36 [10:06:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.97, 2.85, 3.19 [10:07:04] It's a MariaDB bug [10:07:16] But Wikimedia care because they are going 10.4 -> 10.6 [10:07:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 11.71, 10.68 [10:09:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 11.18, 10.59 [10:11:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.90, 9.84, 10.16 [10:12:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.45, 3.28 [10:16:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 3.87, 3.46 [10:17:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 7.00, 6.46, 3.28 [10:18:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.67, 3.30, 3.29 [10:25:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.31, 3.19 [10:27:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.39, 3.22 [10:28:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.49, 3.11 [10:30:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.62, 2.73, 2.87 [10:31:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.10, 2.88, 3.05 [10:33:00] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.16, 1.35, 1.91 [10:37:01] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.62, 0.93, 1.61 [10:39:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.65, 3.33 [10:41:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.40, 3.23, 3.22 [10:42:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.06, 11.22, 9.40 [10:44:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.62, 10.31, 9.30 [10:46:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.99, 9.65, 9.16 [10:47:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.56, 3.32 [10:49:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.42, 3.29 [10:55:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.31, 3.33, 3.30 [11:02:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.51, 3.41 [11:05:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 3.31, 3.37 [11:22:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.05, 4.28, 3.23 [11:23:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.35, 10.18, 9.14 [11:24:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.85, 3.20 [11:25:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.09, 10.78, 9.48 [11:26:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 3.26, 3.05 [11:28:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.80, 10.29, 9.45 [11:29:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 10.54, 9.73 [11:30:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.07, 9.64, 9.31 [11:31:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.46, 10.14, 9.69 [11:33:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.01, 6.38, 5.39 [11:35:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.29, 11.10, 10.15 [11:35:24] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.71, 6.52, 5.57 [11:37:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.99, 10.74, 10.15 [11:39:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.93, 9.90, 9.93 [11:48:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.30, 2.93 [11:49:13] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.64, 7.04, 6.23 [11:50:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.39, 3.01 [11:50:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.35, 2.92 [11:50:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.34, 2.99 [11:51:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.68, 10.55, 10.05 [11:52:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.35, 3.25, 2.93 [11:52:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.86, 3.23 [11:53:11] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.56, 7.90, 6.75 [11:54:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 2.99, 2.95 [11:54:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.78, 3.28 [11:56:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.40, 3.20 [11:57:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.72, 7.37, 6.81 [11:57:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.75, 10.92, 10.30 [11:59:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.87, 11.41, 10.57 [12:01:06] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.04, 6.55, 6.63 [12:02:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.12, 10.11, 9.29 [12:04:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.72, 9.05, 9.00 [12:10:42] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.591 second response time [12:10:42] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:04] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:07] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:11] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:16] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:18] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:20] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.79, 10.28, 9.44 [12:14:19] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.34, 2.70, 1.86 [12:14:51] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.549 second response time [12:14:53] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 9.386 second response time [12:15:01] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.676 second response time [12:15:03] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [12:15:12] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.063 second response time [12:15:12] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.312 second response time [12:15:18] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 27065 bytes in 0.986 second response time [12:15:19] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.218 second response time [12:16:19] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.77, 2.00, 1.70 [12:17:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.02, 9.69, 9.43 [12:18:19] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.46, 1.47, 1.54 [12:19:34] PROBLEM - wiki.evscope.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.evscope.org' expires in 15 day(s) (Mon 25 Jul 2022 12:13:51 GMT +0000). [12:21:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 11.47, 5.97, 4.07 [12:21:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dc59e39b15c5...b32102ceec27 [12:21:58] [url] Comparing dc59e39b15c5...b32102ceec27 · miraheze/ssl · GitHub | github.com [12:21:59] [02miraheze/ssl] 07MirahezeSSLBot 03b32102c - Bot: Update SSL cert for wiki.evscope.org [12:22:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.87, 3.91, 2.97 [12:22:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.83, 5.59, 3.88 [12:23:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.14, 2.22, 1.64 [12:23:04] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:00] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.01, 1.75, 1.53 [12:25:02] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.336 second response time [12:26:31] PROBLEM - lgbtqia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lgbtqia.wiki' expires in 15 day(s) (Mon 25 Jul 2022 12:23:36 GMT +0000). [12:27:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.47, 2.05, 1.67 [12:27:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.38, 11.53, 11.07 [12:29:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.48, 11.41, 11.08 [12:29:40] PROBLEM - www.lgbtqia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lgbtqia.wiki' expires in 15 day(s) (Mon 25 Jul 2022 12:23:36 GMT +0000). [12:29:55] PROBLEM - wikicompliance.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikicompliance.com' expires in 15 day(s) (Mon 25 Jul 2022 12:20:24 GMT +0000). [12:31:00] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.18, 1.67, 1.60 [12:35:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b32102ceec27...2c6065021b33 [12:35:58] [url] Comparing b32102ceec27...2c6065021b33 · miraheze/ssl · GitHub | github.com [12:35:59] [02miraheze/ssl] 07MirahezeSSLBot 032c60650 - Bot: Update SSL cert for www.lgbtqia.wiki [12:37:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.34, 11.07 [12:37:38] PROBLEM - legacygt.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'legacygt.wiki' expires in 15 day(s) (Mon 25 Jul 2022 12:34:19 GMT +0000). [12:38:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [12:38:11] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.44, 2.80, 1.71 [12:38:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.70, 10.55, 9.71 [12:39:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.43, 11.29, 11.08 [12:39:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.37, 10.34, 9.49 [12:39:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2c6065021b33...ac1ffde1f181 [12:39:37] [url] Comparing 2c6065021b33...ac1ffde1f181 · miraheze/ssl · GitHub | github.com [12:39:38] [02miraheze/ssl] 07MirahezeSSLBot 03ac1ffde - Bot: Update SSL cert for wikicompliance.com [12:40:11] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 1.44, 2.32, 1.67 [12:41:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ac1ffde1f181...c0eba30d9f6d [12:41:40]