[00:02:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset -0.119527638 secs [00:07:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.65, 4.72, 3.78 [00:08:02] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.32, 3.96, 3.89 [00:11:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.49, 3.63, 3.72 [00:13:47] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.38, 3.09, 3.51 [00:15:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.42, 2.84, 3.37 [00:20:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.06557261944 secs [00:22:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.96, 3.47, 3.46 [00:25:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.21, 2.86, 3.81 [00:26:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.50, 3.89, 3.60 [00:28:15] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.80, 3.63, 3.53 [00:29:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.81, 2.22, 3.32 [00:30:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.60, 2.93, 3.29 [00:38:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.29, 4.86, 3.97 [00:38:54] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1338522732 secs [00:46:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.208906889 secs [00:47:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.73, 3.21, 3.84 [00:50:54] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.07897877693 secs [00:55:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.84, 2.88, 3.37 [00:55:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 13.33, 6.99, 4.44 [01:03:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.68, 3.25, 3.67 [01:07:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.37, 4.27, 3.45 [01:07:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.31, 3.19, 3.50 [01:08:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.02398404479 secs [01:09:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.90, 3.35, 3.21 [01:09:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.35, 2.77, 3.30 [01:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.23, 3.86, 3.19 [01:31:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.75, 3.69, 3.37 [01:33:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.19, 3.24, 3.25 [01:36:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.23, 4.79, 3.56 [01:42:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset -0.1159259677 secs [01:46:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.09008565545 secs [01:49:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.64, 2.99, 3.91 [01:51:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.23, 4.76, 3.59 [01:55:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.06, 2.22, 3.33 [01:55:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.78, 3.98, 3.57 [01:57:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.17, 3.34, 3.38 [02:07:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.51, 3.49, 3.26 [02:17:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.22, 3.96, 3.94 [02:21:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.80, 4.73, 3.43 [02:23:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.54, 3.90, 3.29 [02:25:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.47, 2.80, 3.34 [02:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.36, 4.28, 3.50 [02:27:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.08, 7.39, 5.75 [02:27:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.41, 7.80, 5.92 [02:29:00] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.42, 6.45, 5.60 [02:29:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.75, 6.88, 5.81 [02:29:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.16, 3.27, 3.28 [02:31:10] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.58, 6.11, 5.66 [02:41:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.87, 3.42, 3.31 [02:43:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.11, 2.95, 3.14 [02:53:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 2.91, 2.58 [02:59:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.14, 2.76, 2.61 [03:02:43] PROBLEM - db131 Backups SQL on db131 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864162 seconds old and 0 bytes [03:03:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.50, 3.70, 3.01 [03:05:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.78, 2.95, 2.82 [03:08:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.89, 3.78, 3.00 [03:10:07] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.04, 4.11, 3.22 [03:13:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.00, 3.43, 3.18 [03:15:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.00, 2.56, 2.88 [03:56:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.45, 3.13, 2.70 [04:01:50] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.32, 2.70, 2.74 [04:05:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.70, 2.68, 2.11 [04:07:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.23, 2.60, 2.16 [04:26:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.45, 3.23, 2.51 [04:30:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.47, 3.05, 2.62 [04:31:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.41, 2.82, 1.59 [04:33:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.99, 2.60, 1.67 [04:39:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.90, 4.67, 3.06 [04:44:32] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [04:48:32] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [04:55:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.65, 3.05, 2.27 [04:57:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.98, 3.14, 3.99 [04:57:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 4.00, 3.21, 2.42 [04:59:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.94, 2.85, 2.39 [05:03:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.93, 4.12, 4.07 [05:06:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.98, 3.49, 2.94 [05:07:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.56, 3.79, 3.98 [05:08:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.15, 3.38, 2.97 [05:11:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.73, 4.44, 4.18 [05:13:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.53, 3.20, 3.75 [05:17:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.95, 2.52, 3.30 [05:25:41] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.90, 3.17, 2.67 [05:27:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.95, 2.97, 2.65 [05:35:57] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.43, 5.28, 2.85 [05:37:52] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.40, 3.99, 2.67 [05:39:48] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.58, 2.80, 2.38 [06:00:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1425322294 secs [06:04:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08909156919 secs [06:27:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.47, 3.05, 2.21 [06:29:10] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.21, 4.21, 2.18 [06:29:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.32, 2.93, 2.29 [06:33:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.83, 3.41, 2.64 [06:35:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.29, 2.67, 2.46 [06:37:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.98, 3.70, 3.10 [06:39:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.96, 2.77, 2.83 [06:44:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.16, 3.39, 2.66 [06:48:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.71, 3.60, 2.92 [06:50:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.09, 3.13, 2.83 [07:04:04] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 15.30, 9.15, 5.23 [07:07:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.53, 4.04, 2.69 [07:13:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.54, 3.67, 3.08 [07:15:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.90, 3.04, 2.92 [07:19:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.14, 2.36, 3.68 [07:21:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.90, 1.87, 3.34 [07:27:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 3.62, 3.67 [07:31:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.87, 2.24, 3.09 [08:01:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.43, 3.73, 2.52 [08:09:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.74, 3.67, 3.61 [08:11:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.67, 2.67, 3.24 [08:37:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.62, 4.22, 3.06 [08:41:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.91, 3.44, 3.05 [08:43:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.07, 2.60, 2.78 [08:59:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.87, 3.69, 2.40 [09:01:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.22, 2.72, 2.20 [09:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.89, 5.16, 3.18 [09:31:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.93, 3.21, 3.10 [09:43:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.63, 4.56, 3.07 [09:51:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.90, 3.22, 3.25 [10:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.59, 3.59, 2.34 [10:27:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.82, 2.67, 2.14 [10:39:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.78, 3.61, 2.17 [10:41:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.81, 3.79, 2.43 [10:45:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.99, 3.35, 2.60 [11:02:01] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.30, 3.52, 2.19 [11:03:57] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.94, 3.43, 2.33 [11:04:15] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.34, 3.80, 2.43 [11:05:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.57, 2.99, 2.31 [11:06:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.30, 2.85, 2.25 [11:26:32] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [11:30:32] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [11:41:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.18, 3.70, 2.50 [11:43:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.10, 3.96, 2.75 [11:47:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.72, 3.48, 2.90 [11:49:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.65, 2.48, 2.60 [11:53:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.94, 2.85, 1.70 [11:55:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.89, 3.72, 2.21 [11:57:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.14, 3.78, 2.41 [11:59:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.09, 3.40, 2.43 [12:13:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.14, 2.95, 2.26 [12:14:35] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.99, 4.36, 2.47 [12:15:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 2.31, 2.10 [12:30:35] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.19, 3.02, 3.83 [12:34:35] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.87, 1.87, 3.17 [12:37:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.52, 3.49, 2.31 [12:39:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.90, 3.78, 2.57 [12:41:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.33, 4.84, 3.08 [12:43:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.43, 3.76, 2.90 [12:45:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.18, 2.86, 2.67 [12:59:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.20, 4.71, 2.72 [13:01:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.92, 3.43, 2.48 [13:03:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.05, 2.59, 2.28 [13:11:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.95, 3.24, 2.41 [13:13:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.58, 3.32, 2.57 [14:07:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.28, 4.79, 3.02 [14:11:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.22, 3.06, 2.72 [14:28:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2254218757 secs [14:34:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.01804065704 secs [14:53:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.32, 3.05, 2.06 [14:55:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.60, 3.02, 2.18 [15:06:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.11, 3.22, 2.40 [15:08:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset -0.1159552336 secs [15:12:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.09573805332 secs [15:29:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.67, 3.72, 2.31 [15:31:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.00, 3.24, 2.31 [15:31:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.51, 2.43, 3.86 [15:39:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.38, 3.32, 2.30 [15:41:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.88, 2.80, 2.24 [15:43:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.33, 2.28, 3.25 [15:55:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.46, 4.97, 3.22 [15:59:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.63, 2.74, 1.97 [16:01:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.14, 2.08, 1.83 [16:01:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.33, 3.28, 3.14 [16:24:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.94, 4.98, 3.27 [16:28:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.40, 3.79, 3.18 [16:30:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.34, 2.95, 2.94 [16:50:51] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [16:55:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.71, 3.85, 2.89 [16:59:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.49, 2.88, 2.74 [17:07:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.98, 3.39, 2.54 [17:09:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.71, 3.18, 2.57 [17:20:11] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:25:45] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 16.50, 8.32, 4.66 [17:37:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.87, 2.90, 2.35 [17:39:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.62, 3.10, 2.52 [17:45:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.70, 2.33, 3.72 [17:49:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.71, 1.99, 3.26 [17:53:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.08, 3.68, 3.66 [18:05:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.49, 2.88, 3.72 [18:09:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.21, 3.62, 3.78 [18:11:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.45, 3.03, 3.54 [18:13:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.91, 2.28, 3.20 [18:18:53] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:25:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.17, 4.81, 3.69 [18:46:35] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.21, 4.37, 2.39 [18:48:35] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.83, 3.46, 2.31 [18:50:35] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.01, 2.64, 2.15 [18:59:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.29, 3.63, 2.82 [19:01:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.65, 3.10, 2.71 [19:05:50] PROBLEM - cp33 PowerDNS Recursor on cp33 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:07:50] RECOVERY - cp33 PowerDNS Recursor on cp33 is OK: DNS OK: 0.199 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [19:16:35] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.48, 3.76, 2.39 [19:19:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/ac5db5ccadaf [19:19:25] [02miraheze/mw-config] 07Universal-Omega 03ac5db5c - Add 1.39 upgrade site notice [19:19:27] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [19:19:30] [02mw-config] 07Universal-Omega opened pull request 03#5067: Add 1.39 upgrade site notice - 13https://github.com/miraheze/mw-config/pull/5067 [19:20:09] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ac5db5ccadaf...1913aad8107d [19:20:10] [02miraheze/mw-config] 07github-actions 031913aad - CI: lint code to MediaWiki standards [19:20:19] [02mw-config] 07github-actions[bot] synchronize pull request 03#5067: Add 1.39 upgrade site notice - 13https://github.com/miraheze/mw-config/pull/5067 [19:20:35] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.38, 4.00, 2.87 [19:20:43] miraheze/mw-config - Universal-Omega the build passed. [19:22:35] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.38, 3.08, 2.67 [19:27:06] [02mw-config] 07Universal-Omega synchronize pull request 03#5067: Add 1.39 upgrade site notice - 13https://github.com/miraheze/mw-config/pull/5067 [19:27:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1913aad8107d...36075b51e310 [19:27:10] [02miraheze/mw-config] 07Universal-Omega 0336075b5 - Update Sitenotice.php [19:28:14] miraheze/mw-config - Universal-Omega the build passed. [19:33:46] [02puppet] 07AgentIsai opened pull request 03#3120: Add GeoIP support for MW IPInfo - 13https://github.com/miraheze/puppet/pull/3120 [19:37:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.98, 3.50, 2.89 [19:38:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [19:41:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.58, 3.06, 2.93 [19:42:04] [02puppet] 07AgentIsai synchronize pull request 03#3120: Add GeoIP support for MW IPInfo - 13https://github.com/miraheze/puppet/pull/3120 [19:45:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.64, 2.72, 3.83 [19:49:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.69, 2.21, 3.36 [19:53:02] [02mw-config] 07Universal-Omega closed pull request 03#5067: Add 1.39 upgrade site notice - 13https://github.com/miraheze/mw-config/pull/5067 [19:53:05] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [19:53:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c402f661b1b2...11b2d87c2a86 [19:53:07] [02miraheze/mw-config] 07Universal-Omega 0311b2d87 - Add 1.39 upgrade site notice (#5067) [19:53:08] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [19:54:08] miraheze/mw-config - Universal-Omega the build passed. [19:55:17] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:55:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:24] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [19:55:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:38] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.33, 3.17, 3.42 [19:56:38] !log [@test131] starting deploy of {'config': True} to all [19:56:39] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:56:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.30, 2.95, 3.32 [19:59:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.25, 3.30, 2.12 [20:06:31] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:11:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.85, 3.19, 3.31 [20:21:03] [02puppet] 07AgentIsai synchronize pull request 03#3120: Add GeoIP support for MW IPInfo - 13https://github.com/miraheze/puppet/pull/3120 [20:22:32] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [20:26:32] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [20:29:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.41, 4.02, 3.06 [20:30:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1526952088 secs [20:31:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.30, 2.97, 2.79 [20:35:50] PROBLEM - itg.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for itg.wiki could not be found [20:38:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.05997312069 secs [20:56:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.06, 5.30, 3.80 [21:04:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.38, 3.24, 3.54 [21:05:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.94, 2.57, 3.26 [21:08:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.64, 3.77, 2.97 [21:09:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.70, 3.76, 3.54 [21:10:06] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.21, 3.27, 2.89 [21:15:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.06, 3.84, 3.81 [21:15:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.86, 5.07, 3.74 [21:19:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.34, 2.61, 3.32 [21:21:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.25, 3.37, 3.41 [21:23:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.33, 3.03, 3.27 [21:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.42, 3.62, 3.54 [21:27:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.48, 3.25, 3.42 [21:29:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.38, 2.62, 3.17 [22:00:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.74, 3.73, 2.94 [22:02:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1061789393 secs [22:05:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.33, 5.39, 3.72 [22:18:17] [02puppet] 07AgentIsai synchronize pull request 03#3120: Add GeoIP support for MW IPInfo - 13https://github.com/miraheze/puppet/pull/3120 [22:18:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09842947125 secs [22:21:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.12, 2.88, 3.82 [22:22:06] [02puppet] 07AgentIsai synchronize pull request 03#3120: Add GeoIP support for MW IPInfo - 13https://github.com/miraheze/puppet/pull/3120 [22:25:50] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.13, 4.44, 3.28 [22:27:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.43, 2.25, 3.26 [22:37:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.35, 4.59, 3.71 [22:37:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.61, 3.67, 3.93 [22:38:34] [02puppet] 07redbluegreenhat opened pull request 03#3121: T10285: Set Content-Disposition header on static.miraheze.org requests containing ?download - 13https://github.com/miraheze/puppet/pull/3121 [22:40:46] [02puppet] 07AgentIsai synchronize pull request 03#3120: Add GeoIP support for MW IPInfo - 13https://github.com/miraheze/puppet/pull/3120 [22:41:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.68, 3.73, 3.56 [22:41:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.87, 2.57, 3.40 [22:41:59] [02puppet] 07AgentIsai synchronize pull request 03#3120: Add GeoIP support for MW IPInfo - 13https://github.com/miraheze/puppet/pull/3120 [22:43:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.60, 3.96, 3.65 [22:45:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.49, 3.50 [22:47:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.86, 4.01, 3.70 [22:48:32] [02mw-config] 07AgentIsai opened pull request 03#5068: T9503: Enable IPInfo globally - 13https://github.com/miraheze/mw-config/pull/5068 [22:49:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.34, 3.34, 3.49 [22:49:22] [02mw-config] 07AgentIsai synchronize pull request 03#5068: T9503: Enable IPInfo globally - 13https://github.com/miraheze/mw-config/pull/5068 [22:49:35] miraheze/mw-config - AgentIsai the build passed. [22:49:38] [02mw-config] 07Universal-Omega commented on pull request 03#5068: T9503: Enable IPInfo globally - 13https://github.com/miraheze/mw-config/pull/5068#issuecomment-1382954079 [22:50:21] miraheze/mw-config - AgentIsai the build has errored. [22:53:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.27, 2.50, 3.13 [22:53:59] [02mw-config] 07AgentIsai synchronize pull request 03#5068: T9503: Enable IPInfo globally - 13https://github.com/miraheze/mw-config/pull/5068 [22:55:00] miraheze/mw-config - AgentIsai the build has errored. [22:55:17] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.01, 3.89, 3.57 [22:56:31] [02mw-config] 07AgentIsai synchronize pull request 03#5068: T9503: Enable IPInfo globally - 13https://github.com/miraheze/mw-config/pull/5068 [22:57:12] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.67, 3.39, 3.42 [22:57:23] miraheze/mw-config - AgentIsai the build passed. [22:59:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.17, 2.65, 3.15 [22:59:59] [02mw-config] 07AgentIsai commented on pull request 03#5068: T9503: Enable IPInfo globally - 13https://github.com/miraheze/mw-config/pull/5068#issuecomment-1382955693 [23:06:26] [02puppet] 07AgentIsai synchronize pull request 03#3120: Add GeoIP support for MW IPInfo - 13https://github.com/miraheze/puppet/pull/3120 [23:10:18] [02puppet] 07AgentIsai synchronize pull request 03#3120: Add GeoIP support for MW IPInfo - 13https://github.com/miraheze/puppet/pull/3120 [23:10:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2178687155 secs [23:10:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.77, 4.18, 3.53 [23:16:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.79, 3.63, 3.53 [23:18:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.21, 2.77, 3.22 [23:21:11] PROBLEM - www.arru.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query arru.xyz. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:24:08] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.51, 4.46, 3.21 [23:26:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.0002054274082 secs [23:29:53] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.99, 3.24, 3.21 [23:39:37] [02puppet] 07AgentIsai opened pull request 03#3122: Automatically update StopForumSpam list - 13https://github.com/miraheze/puppet/pull/3122 [23:40:25] [02puppet] 07AgentIsai synchronize pull request 03#3122: Automatically update StopForumSpam list - 13https://github.com/miraheze/puppet/pull/3122 [23:45:14] [02puppet] 07AgentIsai synchronize pull request 03#3122: Automatically update StopForumSpam list - 13https://github.com/miraheze/puppet/pull/3122 [23:50:52] RECOVERY - www.arru.xyz - reverse DNS on sslhost is OK: SSL OK - www.arru.xyz reverse DNS resolves to cp22.miraheze.org - NS RECORDS OK