[00:01:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.28, 2.24, 1.95 [00:01:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.56, 1.86, 1.88 [00:01:50] RECOVERY - es131 NTP time on es131 is OK: NTP OK: Offset 0.00232821703 secs [00:03:56] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 3873 MB (10% inode=96%); [00:05:56] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 10188 MB (26% inode=96%); [00:07:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.61, 4.25, 5.98 [00:11:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.32, 1.86, 1.94 [00:13:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.55, 5.31, 5.73 [00:15:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.63, 5.64, 5.83 [00:17:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.43, 7.12, 6.35 [00:21:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 1.84, 1.86 [00:25:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.70, 3.12 [00:27:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 3.38, 3.07 [00:29:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 1.76, 1.89 [00:32:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.93, 1.75, 2.00 [00:34:56] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.60, 1.92, 2.03 [00:36:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.44, 3.23 [00:37:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.56, 1.53, 1.70 [00:38:53] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.08, 1.61, 1.88 [00:40:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.53, 3.27 [00:42:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.39, 3.34, 3.23 [00:44:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.19, 1.37, 1.69 [00:46:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.74, 4.24, 3.61 [00:47:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.78, 1.72 [00:49:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.56, 2.15, 1.87 [00:50:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.75, 3.30, 3.40 [00:50:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.10, 2.59, 2.08 [00:54:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.35, 1.99, 1.95 [00:57:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.16, 1.88, 1.91 [01:02:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.10, 1.41, 1.68 [01:07:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.80, 1.91, 1.86 [01:07:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.00, 3.33, 2.95 [01:10:27] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.30, 2.74, 2.19 [01:13:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.65, 3.27 [01:15:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.93, 1.93 [01:15:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.91, 4.33, 5.93 [01:15:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.49, 3.21, 3.15 [01:19:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 1.89, 1.90 [01:19:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.72, 3.18, 5.09 [01:21:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.73, 3.36 [01:23:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.62, 1.95, 1.93 [01:23:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.62, 3.36 [01:25:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 3.33, 3.29 [01:29:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.05, 1.40, 1.69 [01:29:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.99, 8.14, 6.19 [01:30:00] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 15.26, 8.56, 4.60 [01:30:00] PROBLEM - db101 APT on db101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [01:32:05] RECOVERY - db101 APT on db101 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [01:37:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.93, 2.11, 1.83 [01:37:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.87, 5.45, 5.83 [01:39:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.74, 3.41 [01:39:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.34, 6.95, 6.32 [01:40:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.53, 3.39 [01:41:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.29, 5.46, 5.85 [01:42:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 3.26, 3.32 [01:43:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.82, 1.82 [01:43:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.95, 6.21, 6.05 [01:43:59] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.84, 7.96, 6.98 [01:45:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.75, 5.11, 5.66 [01:45:59] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.09, 7.94, 7.08 [01:46:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.74, 1.99 [01:47:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.40, 1.65 [01:47:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.75, 6.43, 6.08 [01:48:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.03, 1.92, 2.03 [01:51:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.58, 2.46, 1.98 [01:51:59] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.53, 7.06, 7.15 [01:53:59] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 1.29, 5.04, 6.40 [01:57:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.89, 1.89 [01:58:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.89, 1.60, 1.92 [01:59:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 1.99, 1.93 [02:03:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.96, 1.95 [02:04:39] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [02:05:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.01, 1.97, 1.95 [02:07:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.63, 4.46, 5.93 [02:13:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.14, 3.12, 4.83 [02:13:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.64, 3.32 [02:15:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.36, 3.32, 3.23 [02:19:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.84, 1.96 [02:21:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 2.05, 2.03 [02:26:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.21, 1.87, 1.85 [02:27:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.81, 1.98 [02:27:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.95, 7.23, 5.75 [02:28:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.31, 1.68, 1.78 [02:33:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.25, 1.66 [02:34:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.35, 2.05, 1.87 [02:34:34] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [02:36:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.39, 1.82, 1.82 [02:37:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.73, 2.04, 1.89 [02:38:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.86, 1.49, 1.69 [02:45:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.44, 1.88, 1.92 [02:45:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.13, 1.90, 1.82 [02:47:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.53, 1.70, 1.75 [02:49:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.13, 1.70, 1.82 [02:49:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.43, 2.02, 1.86 [02:51:14] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.77, 1.99, 1.87 [02:57:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.37, 1.78, 1.78 [02:57:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.09, 4.65, 5.86 [02:59:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.49, 1.63, 1.72 [03:01:06] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 1.84, 1.78 [03:05:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.17, 1.94, 1.89 [03:05:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.61, 1.88, 1.97 [03:05:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.69, 3.35, 4.77 [03:07:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.61, 2.24, 2.01 [03:07:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.73, 2.06, 2.02 [03:09:00] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.35, 2.00, 1.95 [03:09:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.61, 1.90, 1.97 [03:10:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.40, 2.54, 2.16 [03:11:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.76, 2.63, 2.19 [03:13:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.55, 2.45, 3.75 [03:15:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.40, 3.86, 4.12 [03:17:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.25, 1.82, 1.95 [03:17:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.16, 4.84, 4.56 [03:19:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.67, 4.34, 4.41 [03:23:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.79, 5.40, 4.91 [03:25:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 1.99, 1.92 [03:25:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.39, 4.39, 4.61 [03:27:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.07, 1.58, 1.78 [03:29:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.22, 2.99, 3.91 [03:31:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.94, 3.94, 4.11 [03:31:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 1.72, 1.78 [03:33:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.41, 3.90 [03:37:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.30, 2.44, 3.40 [03:37:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.50, 1.67 [03:47:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.94, 1.77 [03:53:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.46, 3.35 [03:55:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.96, 4.30, 3.66 [03:55:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.46, 1.90, 1.78 [03:57:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.22, 1.60, 1.68 [04:00:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.66, 1.82, 1.98 [04:01:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.95, 6.01, 4.71 [04:03:02] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.53, 5.97, 4.85 [04:03:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.44, 2.25, 1.90 [04:07:00] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.74, 6.34, 5.18 [04:08:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.71, 1.23, 1.65 [04:08:59] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.50, 5.18, 4.90 [04:10:58] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.92, 4.12, 4.55 [04:13:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.57, 3.52, 3.93 [04:13:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.72, 1.76, 1.90 [04:14:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.99, 3.04, 2.29 [04:15:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.98, 4.31, 4.13 [04:17:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.73, 3.94 [04:21:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.47, 2.51, 3.38 [04:21:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.86, 2.46, 2.09 [04:30:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.36, 1.72, 1.96 [04:32:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 1.89, 1.99 [04:34:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.79, 1.93, 1.99 [04:36:51] PROBLEM - atnarsia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - atnarsia.com All nameservers failed to answer the query. [04:38:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.45, 3.71 [04:38:40] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.31, 5.10, 4.03 [04:39:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.81, 1.96 [04:40:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.17, 1.88, 1.90 [04:40:39] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.07, 3.94, 3.74 [04:41:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.75, 4.20, 2.88 [04:42:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.66, 1.76, 1.85 [04:44:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.29, 1.90, 1.89 [04:49:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [04:51:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:55:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.84, 3.49, 3.87 [04:57:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [05:01:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.74, 2.30, 3.21 [05:05:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.64, 1.57, 1.96 [05:05:41] RECOVERY - atnarsia.com - reverse DNS on sslhost is OK: SSL OK - atnarsia.com reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [05:07:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.55, 1.79, 2.01 [05:09:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.69, 1.94 [05:09:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:19:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.55, 1.77, 1.81 [05:23:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.37, 6.35, 4.38 [05:23:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 4.46, 3.56 [05:25:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.85, 5.52, 4.32 [05:29:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.83, 3.98, 3.73 [05:29:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.47, 4.34, 4.16 [05:31:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.87, 1.68, 1.88 [05:33:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.82, 2.22, 2.04 [05:35:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 10.56, 5.00, 4.02 [05:39:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.75, 5.57, 4.63 [05:41:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.87, 1.96 [05:41:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.96, 4.44, 4.33 [05:41:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [05:42:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.78, 1.37, 1.97 [05:47:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.70, 5.60, 4.96 [05:49:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.47, 4.82, 4.75 [05:51:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.08, 3.67, 3.98 [05:52:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.04, 1.24, 1.63 [05:55:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.28, 1.34, 1.64 [05:56:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.59, 1.71 [05:58:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 2.06, 1.88 [05:59:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.40, 3.93, 3.84 [06:01:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.86, 3.83 [06:01:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:02:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.59, 1.97, 1.90 [06:03:39] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:04:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.04, 2.27, 2.01 [06:05:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.67, 4.59, 4.12 [06:05:50] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36229 bytes in 1.057 second response time [06:05:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.52, 1.95, 1.76 [06:08:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.53, 1.86, 1.90 [06:09:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 2.00, 1.83 [06:13:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:13:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.92, 2.08, 1.91 [06:14:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 1.89, 1.88 [06:17:24] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.53, 6.87, 5.23 [06:23:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.42, 5.26, 5.25 [06:25:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.88, 4.26, 4.88 [06:25:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.90, 1.96 [06:31:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.98, 2.48, 3.77 [06:34:46] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:35:17] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.10, 2.10, 1.90 [06:36:48] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 7.329 second response time [06:37:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.07, 3.57 [06:37:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.23, 1.76, 1.80 [06:43:07] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.26, 2.32, 2.00 [06:44:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.93, 1.36, 1.92 [06:47:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.80, 1.87 [06:47:44] PROBLEM - cp20 APT on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:48:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.16, 1.71, 1.92 [06:49:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.81, 3.88 [06:50:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.58, 1.62, 1.86 [06:53:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:55:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.25, 1.81, 1.83 [06:55:33] RECOVERY - cp20 APT on cp20 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [06:56:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.94, 1.30, 1.64 [06:57:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.44, 2.50, 3.24 [07:05:06] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:07:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.75, 1.63, 1.86 [07:09:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.02, 2.00, 1.95 [07:09:54] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 0.081 second response time [07:11:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.99, 3.59, 3.11 [07:11:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.89, 1.92 [07:13:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.91, 3.28 [07:14:02] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 12.98, 6.16, 4.14 [07:15:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.35, 1.69 [07:15:53] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:17:48] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 1.062 second response time [07:18:00] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.82, 4.91, 4.13 [07:18:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cba50641c0fe...f359ac025eaf [07:18:53] [url] Comparing cba50641c0fe...f359ac025eaf · miraheze/mw-config · GitHub | github.com [07:18:54] [02miraheze/mw-config] 07Universal-Omega 03f359ac0 - Disable WikiForum [07:19:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.84, 3.54 [07:19:50] miraheze/mw-config - Universal-Omega the build passed. [07:21:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.20, 2.16, 1.90 [07:23:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 3.12, 3.32 [07:25:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.83, 1.84 [07:26:26] !log [@test131] starting deploy of {'config': True} to all [07:26:27] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [07:26:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:26:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.62, 2.15, 1.84 [07:29:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.77, 1.31, 1.63 [07:30:22] PROBLEM - mwtask141 Disk Space on mwtask141 is CRITICAL: DISK CRITICAL - free space: / 3650 MB (5% inode=90%); [07:33:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.61, 1.82, 1.78 [07:37:54] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.11, 1.52, 1.67 [07:38:22] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 35182 MB (49% inode=90%); [07:40:12] !log [@mwtask141] starting deploy of {'config': True} to all [07:40:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:40:21] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [07:40:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:51:14] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.72, 2.29, 1.82 [07:55:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.89, 1.79 [07:55:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:57:07] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.78, 2.26, 1.93 [08:05:10] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f359ac025eaf...80a311f06954 [08:05:10] [url] Comparing f359ac025eaf...80a311f06954 · miraheze/mw-config · GitHub | github.com [08:05:11] [02miraheze/mw-config] 07Reception123 0380a311f - wikiforum disable notice [08:05:23] !log [reception@mwtask141] starting deploy of {'config': True} to all [08:05:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk [08:05:31] !log [reception@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [08:05:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:06:10] miraheze/mw-config - Reception123 the build passed. [08:07:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.86, 1.94 [08:07:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.23, 2.08, 1.78 [08:09:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.73, 2.29, 2.10 [08:10:21] !log [@mwtask141] starting deploy of {'config': True} to all [08:10:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:10:29] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [08:10:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:11:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.06, 1.87, 1.97 [08:13:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.81, 1.93 [08:15:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.87, 1.94 [08:15:36] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:16:12] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/80a311f06954...7f6426779c38 [08:16:13] [url] Comparing 80a311f06954...7f6426779c38 · miraheze/mw-config · GitHub | github.com [08:16:14] [02miraheze/mw-config] 07Reception123 037f64267 - Update Sitenotice.php [08:16:14] !log [reception@mwtask141] starting deploy of {'config': True} to all [08:16:22] !log [reception@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [08:16:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:16:28] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [08:16:32] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [08:16:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:16:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:16:50] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [08:16:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:17:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:17:15] miraheze/mw-config - Reception123 the build passed. [08:17:33] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 0.055 second response time [08:20:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk [08:21:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.87, 1.88 [08:23:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.12, 2.68 [08:25:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.06, 2.59, 2.53 [08:25:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.92, 1.88, 1.89 [08:27:03] !log [@test131] starting deploy of {'config': True} to all [08:27:04] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [08:27:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:28:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:29:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.03, 4.74, 3.49 [08:29:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 2.06, 1.95 [08:29:15] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.45, 7.08, 4.77 [08:31:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 2.00, 1.95 [08:31:13] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.34, 5.99, 4.65 [08:33:12] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 14.09, 8.00, 5.48 [08:35:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.56, 2.00, 1.94 [08:35:26] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:34] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:37:00] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:37:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.67, 1.85, 1.89 [08:37:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:37:25] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.278 second response time [08:37:32] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.303 second response time [08:38:57] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:39:59] !log [@mwtask141] starting deploy of {'config': True} to all [08:40:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:40:07] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [08:40:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:41:02] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.09, 1.67, 1.93 [08:41:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 1.38, 1.68 [08:41:08] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.56, 5.05, 5.49 [08:43:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.66, 3.61, 3.98 [08:43:07] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.94, 3.99, 5.05 [08:44:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.56, 2.18, 2.04 [08:46:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.54, 1.88, 1.94 [08:48:56] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.71, 2.21, 2.06 [08:49:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.68, 2.24, 3.24 [08:49:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [08:51:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.36, 2.24, 1.85 [08:53:00] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.17, 7.38, 5.64 [08:53:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.79, 3.65 [08:53:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.56, 2.00, 1.82 [08:55:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.66, 3.61 [08:56:58] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.53, 5.32, 5.21 [08:57:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.97, 3.36 [08:57:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.61, 1.69 [08:58:48] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.09, 1.70, 1.94 [08:58:57] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.50, 4.72, 5.00 [09:00:26] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 2.05, 1.85 [09:03:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.91, 1.82 [09:05:18] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 1.061 second response time [09:06:42] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.17, 1.66, 1.78 [09:08:41] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.21, 1.47, 1.69 [09:09:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 2.18, 1.95 [09:11:17] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.25, 1.93, 1.94 [09:16:14] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36229 bytes in 1.163 second response time [09:18:44] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.65, 5.44, 4.54 [09:19:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.62, 3.33 [09:19:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.99, 1.97 [09:19:41] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.19, 1.84, 1.74 [09:20:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk [09:20:43] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.21, 5.39, 4.61 [09:22:27] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.84, 1.85, 1.75 [09:22:41] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 4.19, 5.02, 4.57 [09:25:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.93, 3.15, 3.27 [09:26:24] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.55, 1.60, 1.67 [09:28:12] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:30:11] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 1.151 second response time [09:31:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:33:33] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.45, 6.49, 5.24 [09:35:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.67, 4.46, 3.84 [09:35:32] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.50, 5.14, 4.91 [09:37:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.10, 3.65, 3.62 [09:37:31] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.65, 4.01, 4.53 [09:39:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.57, 2.95, 3.36 [09:40:33] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:43:02] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 3.078 second response time [09:43:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:47:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.15, 1.82, 1.65 [09:48:57] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:51:06] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.89, 1.88, 1.71 [09:57:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.12, 1.95, 1.78 [09:59:00] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.73, 1.88, 1.77 [10:00:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.34, 2.06, 1.85 [10:02:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 3.65, 3.06 [10:06:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.10, 3.30, 3.09 [10:11:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.35, 1.56, 1.91 [10:19:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.12, 2.29, 2.06 [10:21:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:22:02] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.48, 5.32, 4.00 [10:24:01] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 4.27, 4.90, 4.00 [10:24:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.92, 3.30 [10:26:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 10.42, 6.03, 4.11 [10:27:57] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.63, 5.81, 4.60 [10:29:56] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.23, 6.10, 4.85 [10:30:58] !log sudo shutdown on cloud11 (effect at :31:20) [10:31:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:32:34] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.60, 1.73, 2.00 [10:33:54] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.04, 4.59, 4.56 [10:33:54] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [10:34:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:34:33] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.49, 2.03, 2.08 [10:34:47] PROBLEM - ping6 on cloud11 is CRITICAL: PING CRITICAL - Packet loss = 100% [10:35:04] PROBLEM - Host cloud11 is DOWN: PING CRITICAL - Packet loss = 100% [10:37:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [10:40:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk [10:43:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.92, 3.37, 3.83 [10:43:59] PROBLEM - cp20 APT on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:46:24] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.85, 1.59, 1.98 [10:46:46] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:47:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:48:36] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [10:48:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:48:53] RECOVERY - cp20 APT on cp20 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [10:49:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.51, 2.29, 3.20 [10:50:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.24, 2.05, 2.07 [10:51:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:51:42] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 1.069 second response time [10:52:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.00, 1.77, 1.97 [10:58:38] PROBLEM - cloud14 IPMI Sensors on cloud14 is CRITICAL: IPMI Status: Critical [3 system event log (SEL) entries present] [11:01:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.69, 1.45, 1.92 [11:01:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.38, 3.22 [11:01:51] PROBLEM - lore.checkit.ws - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lore.checkit.ws All nameservers failed to answer the query. [11:03:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.50, 4.24, 3.54 [11:04:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.01, 1.85, 1.87 [11:05:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.91, 3.50 [11:06:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.16, 1.59, 1.77 [11:07:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.81, 3.13, 3.26 [11:09:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.61, 1.80, 1.85 [11:10:52] PROBLEM - cloud13 IPMI Sensors on cloud13 is CRITICAL: IPMI Status: Critical [3 system event log (SEL) entries present] [11:11:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.87, 1.88, 1.88 [11:12:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.17, 1.89, 1.84 [11:13:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 2.04, 1.94 [11:13:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:14:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.64, 1.82, 1.82 [11:18:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.45, 1.79, 1.79 [11:23:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.64, 3.35 [11:24:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.17, 1.76, 1.81 [11:25:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.19, 3.21 [11:27:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:28:19] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.29, 5.60, 3.96 [11:28:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.25, 2.36, 2.02 [11:29:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 3.08, 2.24 [11:29:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 10.55, 6.23, 4.38 [11:30:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.05, 1.83, 1.87 [11:31:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.88, 2.91, 2.27 [11:31:51] PROBLEM - lore.checkit.ws - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lore.checkit.ws could not be found [11:34:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.24, 2.02, 1.94 [11:36:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.87, 1.89 [11:38:13] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.89, 5.16, 4.96 [11:39:19] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 0.55 ms [11:39:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:39:51] RECOVERY - ping6 on cloud11 is OK: PING OK - Packet loss = 0%, RTA = 1.20 ms [11:40:12] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.08, 6.22, 5.32 [11:41:08] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 5 = Critical, Cntlr 2 Bay 6 = Critical, Cntlr 2 Bay 7 = Critical] [11:41:29] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,4] - No health status line found, Self-test log contains errors --- [cciss,5] - No health status line found, Self-test log contains errors --- [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean| [11:41:55] PROBLEM - bast101 Current Load on bast101 is WARNING: WARNING - load average: 1.84, 1.10, 0.44 [11:42:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.15, 2.21, 1.98 [11:43:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:43:49] RECOVERY - bast101 Current Load on bast101 is OK: OK - load average: 0.27, 0.75, 0.38 [11:43:53] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_landing],Exec[git_pull_ErrorPages],Exec[git_pull_JobRunner] [11:46:09] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.83, 5.85, 5.70 [11:50:07] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.14, 5.92, 5.60 [12:03:23] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:05:49] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 0.048 second response time [12:09:51] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [12:10:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.20, 1.78, 1.97 [12:11:54] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.38, 4.34, 5.72 [12:18:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [12:19:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.29, 1.54, 1.99 [12:19:49] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.54, 6.17, 5.83 [12:21:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.68, 1.85, 2.04 [12:24:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.52, 1.46, 1.69 [12:35:18] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.81, 1.74, 1.69 [12:37:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.37, 2.29, 1.89 [12:43:36] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.99, 4.29, 5.77 [12:45:10] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.37, 1.93, 1.94 [12:49:32] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 11.40, 5.87, 5.72 [12:49:57] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:51:06] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.24, 1.43, 1.70 [12:52:08] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36267 bytes in 1.159 second response time [12:56:01] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:57:49] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.76, 1.56, 1.25 [12:59:49] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.28, 1.48, 1.26 [12:59:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [13:03:45] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:06:50] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 1.85, 1.71 [13:07:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.52, 4.93, 5.93 [13:13:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.97, 3.75, 5.04 [13:13:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [13:16:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.50, 1.87, 1.83 [13:17:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.89, 4.77, 5.26 [13:18:41] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.86, 2.18, 1.94 [13:19:27] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.85, 6.11, 5.65 [13:22:31] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:23:25] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.70, 5.72, 5.65 [13:27:19] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36229 bytes in 0.053 second response time [13:27:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.12, 3.85, 4.90 [13:31:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.37, 5.80, 5.46 [13:42:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.31, 9.78, 8.07 [13:43:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.28, 4.85, 5.82 [13:44:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.34, 9.19, 8.06 [13:49:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:49:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.03, 6.11, 5.86 [13:53:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.09, 4.81, 5.41 [13:54:56] PROBLEM - sharqedia.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sharqedia.ml' expires in 15 day(s) (Mon 03 Oct 2022 13:40:45 GMT +0000). [13:55:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.33, 3.94, 5.02 [13:55:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/92bc8c7937dc...5d20bf7cc034 [13:55:55] [url] Comparing 92bc8c7937dc...5d20bf7cc034 · miraheze/ssl · GitHub | github.com [13:55:56] [02miraheze/ssl] 07MirahezeSSLBot 035d20bf7 - Bot: Update SSL cert for sharqedia.ml [14:02:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.67, 1.68, 1.98 [14:03:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:03:50] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:06:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 1.90, 1.98 [14:08:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.55, 1.85, 1.96 [14:09:36] PROBLEM - wiki.fatcatfablab.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fatcatfablab.org' expires in 15 day(s) (Mon 03 Oct 2022 13:44:30 GMT +0000). [14:09:58] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 2.078 second response time [14:10:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.82, 2.76, 2.29 [14:11:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.04, 2.90, 3.79 [14:12:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5d20bf7cc034...89e7ca5448fe [14:12:09] [url] Comparing 5d20bf7cc034...89e7ca5448fe · miraheze/ssl · GitHub | github.com [14:12:10] [02miraheze/ssl] 07MirahezeSSLBot 0389e7ca5 - Bot: Update SSL cert for wiki.fatcatfablab.org [14:19:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.65, 4.34, 3.94 [14:22:39] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [14:24:44] RECOVERY - cp30 APT on cp30 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [14:26:05] PROBLEM - cp20 APT on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:28:03] RECOVERY - cp20 APT on cp20 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [14:28:19] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.82, 7.04, 5.01 [14:38:13] RECOVERY - wiki.fatcatfablab.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fatcatfablab.org' will expire on Fri 16 Dec 2022 13:12:03 GMT +0000. [14:39:08] PROBLEM - cp20 APT on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:41:06] RECOVERY - cp20 APT on cp20 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [14:44:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.98, 3.36, 2.85 [14:45:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:46:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.00, 2.77, 2.69 [14:52:33] PROBLEM - forecasting.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'forecasting.wiki' expires in 15 day(s) (Mon 03 Oct 2022 14:22:44 GMT +0000). [14:52:56] RECOVERY - sharqedia.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'sharqedia.ml' will expire on Fri 16 Dec 2022 12:55:48 GMT +0000. [14:53:03] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [14:54:03] PROBLEM - cp20 APT on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:57:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:58:56] RECOVERY - cp20 APT on cp20 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [14:59:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/89e7ca5448fe...64acb8f7288a [14:59:05] [url] Comparing 89e7ca5448fe...64acb8f7288a · miraheze/ssl · GitHub | github.com [14:59:05] [02miraheze/ssl] 07MirahezeSSLBot 0364acb8f - Bot: Update SSL cert for forecasting.wiki [14:59:14] RECOVERY - cp30 APT on cp30 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [15:15:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.26, 2.90 [15:17:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.01, 2.89, 2.82 [15:22:06] RECOVERY - forecasting.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'forecasting.wiki' will expire on Fri 16 Dec 2022 13:58:58 GMT +0000. [15:22:56] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:29:12] RECOVERY - cp30 APT on cp30 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [15:31:44] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.80, 4.64, 5.95 [15:33:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.08, 10.36, 8.98 [15:33:43] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.25, 6.08, 6.33 [15:35:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.81, 10.14, 9.07 [15:35:36] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:37:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:37:38] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36267 bytes in 0.076 second response time [15:37:40] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.26, 4.56, 5.72 [15:43:37] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 4.41, 4.07, 5.09 [15:47:34] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.58, 6.27, 5.70 [15:51:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:53:14] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:59:31] RECOVERY - cp30 APT on cp30 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [16:24:08] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [16:26:12] RECOVERY - cp30 APT on cp30 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [16:30:04] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:43:40] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:01:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.05, 4.36, 5.90 [17:07:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.62, 6.68, 6.29 [17:08:03] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:10:05] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36275 bytes in 1.054 second response time [17:13:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.60, 3.01 [17:15:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.13, 2.95, 2.83 [17:23:52] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [17:25:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:27:59] RECOVERY - cp30 APT on cp30 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [17:36:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [17:51:49] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [17:52:00] PROBLEM - cp20 APT on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:54:13] RECOVERY - cp20 APT on cp20 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [17:55:55] RECOVERY - cp30 APT on cp30 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [18:04:37] PROBLEM - intp.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:00] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:34] PROBLEM - test2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:57] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:09:49] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.63, 2.15, 1.57 [18:14:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03recursor [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/461dc4515af8 [18:14:56] [02miraheze/puppet] 07Universal-Omega 03461dc45 - recursor: remove mh and 10.in-addr.arpa from forward-zones [18:14:57] [02puppet] 07Universal-Omega created branch 03recursor - 13https://github.com/miraheze/puppet [18:14:58] [url] Page not found · GitHub · GitHub | github.com [18:14:59] [02puppet] 07Universal-Omega opened pull request 03#2843: recursor: remove mh and 10.in-addr.arpa from forward-zones - 13https://github.com/miraheze/puppet/pull/2843 [18:14:59] [url] Page not found · GitHub · GitHub | github.com [18:17:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:20:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b0510c1fd518...7ffac26092e7 [18:20:44] [url] Comparing b0510c1fd518...7ffac26092e7 · miraheze/puppet · GitHub | github.com [18:20:45] [02miraheze/puppet] 07Universal-Omega 037ffac26 - recursor: remove mh and 10.in-addr.arpa from forward-zones (#2843) [18:20:46] [02puppet] 07paladox closed pull request 03#2843: recursor: remove mh and 10.in-addr.arpa from forward-zones - 13https://github.com/miraheze/puppet/pull/2843 [18:20:47] [url] Page not found · GitHub · GitHub | github.com [18:20:48] [02miraheze/puppet] 07paladox deleted branch 03recursor [18:20:49] [02puppet] 07paladox deleted branch 03recursor - 13https://github.com/miraheze/puppet [18:20:52] [url] Page not found · GitHub · GitHub | github.com [18:26:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 10.22, 5.07, 3.48 [18:27:36] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 5.69, 3.58, 2.11 [18:29:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 9.95, 8.11 [18:29:35] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 2.96, 3.56, 2.30 [18:30:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.60, 3.22 [18:31:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.10, 9.15, 8.04 [18:31:35] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 1.30, 2.76, 2.16 [18:32:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 3.28, 3.15 [18:33:50] RECOVERY - intp.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:34:10] RECOVERY - test2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:51:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [19:05:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:13:49] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 0.99, 1.56, 1.98 [19:19:49] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 0.76, 1.05, 1.61 [19:35:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.26, 4.41, 5.73 [19:41:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.07, 5.12, 5.58 [19:55:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.13, 5.15, 5.81 [20:01:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.97, 5.67, 5.66 [20:03:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.82, 5.09, 5.46 [20:05:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.84, 4.15, 5.06 [20:15:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.38, 5.80, 5.26 [20:18:14] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:23:24] PROBLEM - deepdark.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'deepdark.wiki' expires in 15 day(s) (Mon 03 Oct 2022 20:17:33 GMT +0000). [20:26:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/64acb8f7288a...aa767a1b3f30 [20:26:35] [url] Comparing 64acb8f7288a...aa767a1b3f30 · miraheze/ssl · GitHub | github.com [20:26:36] [02miraheze/ssl] 07MirahezeSSLBot 03aa767a1 - Bot: Update SSL cert for deepdark.wiki [20:48:10] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [21:01:36] PROBLEM - pmwiki.cubestudios.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pmwiki.cubestudios.xyz' expires in 15 day(s) (Mon 03 Oct 2022 20:32:38 GMT +0000). [21:03:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/aa767a1b3f30...df502def738a [21:03:35] [02miraheze/ssl] 07MirahezeSSLBot 03df502de - Bot: Update SSL cert for pmwiki.cubestudios.xyz [21:03:38] [url] Comparing aa767a1b3f30...df502def738a · miraheze/ssl · GitHub | github.com [21:11:29] PROBLEM - wiki.cjgh.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cjgh.xyz' expires in 15 day(s) (Mon 03 Oct 2022 20:57:39 GMT +0000). [21:19:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/df502def738a...fd62f4111ba4 [21:19:11] [url] Comparing df502def738a...fd62f4111ba4 · miraheze/ssl · GitHub | github.com [21:19:12] [02miraheze/ssl] 07MirahezeSSLBot 03fd62f41 - Bot: Update SSL cert for wiki.cjgh.xyz [21:21:24] RECOVERY - deepdark.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deepdark.wiki' will expire on Fri 16 Dec 2022 19:26:28 GMT +0000. [21:21:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.50, 2.92 [21:21:54] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4217 MB (10% inode=96%); [21:23:39] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4223 MB (10% inode=96%); [21:25:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.71, 3.76, 3.14 [21:26:52] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 30.04, 14.76, 8.63 [21:26:55] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:13] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 201 bytes in 0.379 second response time [21:27:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 20.85, 17.81, 11.78 [21:27:18] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:27:25] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [21:27:29] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 29.98, 15.37, 8.82 [21:27:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.39, 3.14, 2.99 [21:27:44] PROBLEM - cp31 HTTPS on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:59] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 201 bytes in 4.074 second response time [21:28:03] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:28:12] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:28:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:28:27] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:28:32] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 21.21, 15.97, 9.73 [21:28:53] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.77, 10.85, 7.68 [21:28:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.47, 14.34, 10.28 [21:29:12] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.239 second response time [21:29:13] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.282 second response time [21:29:20] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.242 second response time [21:29:21] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:29:42] RECOVERY - cp31 HTTPS on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3351 bytes in 0.334 second response time [21:29:58] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.253 second response time [21:29:58] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.236 second response time [21:30:14] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 5.316 second response time [21:30:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:30:23] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.332 second response time [21:30:52] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.01, 9.40, 7.53 [21:30:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 2.91, 9.96, 9.17 [21:30:53] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw122 mw132 [21:30:53] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.248 second response time [21:30:56] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw122 [21:31:28] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 6.97, 11.62, 8.84 [21:31:34] RECOVERY - pmwiki.cubestudios.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'pmwiki.cubestudios.xyz' will expire on Fri 16 Dec 2022 20:03:27 GMT +0000. [21:32:26] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 6.81, 11.17, 9.18 [21:32:46] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 6.18, 11.02, 9.46 [21:32:53] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [21:32:56] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [21:33:29] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.24, 9.83, 8.52 [21:34:24] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 6.01, 9.30, 8.73 [21:34:46] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.74, 9.57, 9.11 [21:35:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.68, 11.11, 10.91 [21:44:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.42, 8.30, 9.66 [21:49:56] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [21:51:10] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [21:55:08] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 5 = Critical, Cntlr 2 Bay 6 = Critical, Cntlr 2 Bay 7 = Critical] [21:55:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.12, 4.02, 3.42 [21:59:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.81, 3.46, 3.38 [22:01:26] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:29] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:32] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:47] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 201 bytes in 0.245 second response time [22:01:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 2.76, 3.12 [22:01:53] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:20] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.195 second response time [22:03:30] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 2.477 second response time [22:03:32] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 5.812 second response time [22:03:49] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 6.651 second response time [22:03:55] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 7.340 second response time [22:04:56] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw142 [22:05:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 4.18, 3.63 [22:07:00] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [22:09:29] RECOVERY - wiki.cjgh.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cjgh.xyz' will expire on Fri 16 Dec 2022 20:19:04 GMT +0000. [22:09:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.37, 3.89, 3.66 [22:11:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.97, 3.71 [22:13:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.19, 3.39, 3.53 [22:17:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 2.92, 3.31 [22:24:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.69, 10.27, 8.29 [22:26:16] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.84, 8.88, 8.02 [22:29:09] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 2.434 second response time [22:29:09] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 39.04, 19.66, 11.00 [22:29:10] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 201 bytes in 8.702 second response time [22:29:11] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:29:16] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:29:20] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: connect to address 2a10:6740::6:309 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:29:22] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 201 bytes in 4.342 second response time [22:29:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.91, 3.96, 5.98 [22:29:42] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 201 bytes in 0.609 second response time [22:29:53] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 1.769 second response time [22:30:16] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 31.06, 19.42, 11.07 [22:30:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.19, 16.94, 11.96 [22:30:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.22, 11.55, 9.53 [22:30:53] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw121 mw131 mw142 [22:30:56] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 3 backends are down. mw121 mw131 mw142 [22:31:25] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:31:28] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 13.97, 16.99, 10.17 [22:31:52] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.431 second response time [22:32:01] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: connect to address 2a10:6740::6:309 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:32:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 0.71, 7.82, 8.41 [22:32:52] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:00] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:04] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: connect to address 2a10:6740::6:403 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:33:11] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.256 second response time [22:33:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.37, 4.65, 5.72 [22:33:24] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.236 second response time [22:33:29] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 2.28, 11.54, 9.00 [22:33:32] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:02] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 0.79, 10.10, 10.13 [22:35:04] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 9.523 second response time [22:35:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.51, 4.23, 5.45 [22:35:28] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.65, 8.58, 8.19 [22:35:29] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 2.543 second response time [22:36:09] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 7.46, 11.98, 10.37 [22:36:52] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.480 second response time [22:38:09] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 6.48, 10.01, 9.84 [22:38:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.47, 11.99, 11.36 [22:41:09] PROBLEM - wiki.recaptime.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.recaptime.eu.org' expires in 15 day(s) (Mon 03 Oct 2022 22:23:06 GMT +0000). [22:41:10] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.276 second response time [22:41:12] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.347 second response time [22:41:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.33, 4.15, 5.09 [22:41:32] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.237 second response time [22:42:01] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.014 second response time [22:42:28] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.356 second response time [22:42:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.41, 11.24, 11.13 [22:42:36] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.249 second response time [22:42:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fd62f4111ba4...6f463aca3f53 [22:42:45] [02miraheze/ssl] 07MirahezeSSLBot 036f463ac - Bot: Update SSL cert for wiki.recaptime.eu.org [22:42:45] [url] Comparing fd62f4111ba4...6f463aca3f53 · miraheze/ssl · GitHub | github.com [22:42:52] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.242 second response time [22:43:04] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [22:44:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.59, 9.65, 10.56 [22:45:56] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.56, 6.82, 5.93 [22:46:53] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [22:46:56] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [22:48:35] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.58, 8.76, 10.04 [22:49:46] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp30.miraheze.org - NS RECORDS OK [22:53:51] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.40, 5.44, 5.83 [22:59:48] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.81, 6.44, 5.93 [23:03:45] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.89, 5.13, 5.53 [23:05:44] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 13.08, 8.17, 6.58 [23:11:06] RECOVERY - wiki.recaptime.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.eu.org' will expire on Fri 16 Dec 2022 21:42:37 GMT +0000. [23:12:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7ffac26092e7...d30af4549870 [23:12:35] [url] Comparing 7ffac26092e7...d30af4549870 · miraheze/puppet · GitHub | github.com [23:12:36] [02miraheze/puppet] 07Universal-Omega 03d30af45 - varnish: make sure match with path_redirects has an ending [23:13:36] [02puppet] 07Universal-Omega closed pull request 03#2838: unattended-upgrades: replace MailOnlyOnError with MailReport - 13https://github.com/miraheze/puppet/pull/2838 [23:13:37] [url] Page not found · GitHub · GitHub | github.com [23:13:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d30af4549870...c9adc8e6f6fe [23:13:38] [url] Comparing d30af4549870...c9adc8e6f6fe · miraheze/puppet · GitHub | github.com [23:13:39] [02miraheze/puppet] 07Universal-Omega 03c9adc8e - unattended-upgrades: replace MailOnlyOnError with MailReport (#2838) [23:13:41] [02puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [23:13:41] [url] Page not found · GitHub · GitHub | github.com [23:13:42] [02miraheze/puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:17:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/b88873773e3a [23:17:37] [02miraheze/mw-config] 07Universal-Omega 03b888737 - Set `$wgCategoryCollation` to `numeric` for `levyraatiwikiwiki` [23:17:39] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [23:17:39] [url] Page not found · GitHub · GitHub | github.com [23:17:40] [02mw-config] 07Universal-Omega opened pull request 03#4902: Set `$wgCategoryCollation` to `numeric` for `levyraatiwikiwiki` - 13https://github.com/miraheze/mw-config/pull/4902 [23:17:41] [url] Page not found · GitHub · GitHub | github.com [23:17:54] [02mw-config] 07Universal-Omega edited pull request 03#4902: T9750: set `$wgCategoryCollation` to `numeric` for `levyraatiwikiwiki` - 13https://github.com/miraheze/mw-config/pull/4902 [23:17:54] [url] Page not found · GitHub · GitHub | github.com [23:18:10] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b88873773e3a...2bac4ce45bd9 [23:18:11] [url] Comparing b88873773e3a...2bac4ce45bd9 · miraheze/mw-config · GitHub | github.com [23:18:12] [02miraheze/mw-config] 07github-actions 032bac4ce - CI: lint code to MediaWiki standards [23:18:13] [02mw-config] 07github-actions[bot] synchronize pull request 03#4902: T9750: set `$wgCategoryCollation` to `numeric` for `levyraatiwikiwiki` - 13https://github.com/miraheze/mw-config/pull/4902 [23:18:14] ... [23:18:44] miraheze/mw-config - Universal-Omega the build passed. [23:20:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7f6426779c38...e628bdcde8ae [23:20:45] [url] Comparing 7f6426779c38...e628bdcde8ae · miraheze/mw-config · GitHub | github.com [23:20:45] [02miraheze/mw-config] 07Universal-Omega 03e628bdc - Bump `$wgMajorSiteNoticeID` [23:21:42] miraheze/mw-config - Universal-Omega the build passed. [23:27:09] !log [@test131] starting deploy of {'config': True} to all [23:27:10] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [23:27:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:32] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.16, 4.52, 5.77 [23:29:31] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.16, 5.32, 5.87 [23:32:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c9adc8e6f6fe...38a9ca6e0159 [23:32:35] [url] Comparing c9adc8e6f6fe...38a9ca6e0159 · miraheze/puppet · GitHub | github.com [23:32:36] [02miraheze/puppet] 07Universal-Omega 0338a9ca6 - blackbox: remove fail_if_body_not_matches_regexp [23:33:29] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.09, 5.02, 5.63 [23:40:24] !log [@mwtask141] starting deploy of {'config': True} to all [23:40:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:36] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [23:40:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:45:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.84, 7.13, 5.99