[00:02:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.33, 4.63, 3.65 [00:03:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.57, 11.59, 11.41 [00:07:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.02, 10.54, 11.14 [00:08:23] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.16, 8.53, 4.92 [00:08:29] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.41, 2.69, 1.63 [00:08:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.13, 4.54, 3.52 [00:09:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.02, 10.99, 11.19 [00:12:27] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.46, 3.91, 2.35 [00:12:47] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.27, 3.34, 3.30 [00:14:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.34, 3.52, 3.98 [00:16:45] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.09, 4.10, 2.31 [00:18:23] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.97, 3.35, 2.70 [00:18:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.62, 3.79, 3.94 [00:20:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.61, 7.67, 7.27 [00:20:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.81, 3.68, 3.90 [00:20:44] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.71, 3.03, 2.39 [00:22:14] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.05, 9.69, 8.06 [00:24:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.30, 4.37, 4.07 [00:26:12] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.04, 6.97, 7.35 [00:26:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.82, 3.70, 3.85 [00:30:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.56, 6.97, 7.18 [00:30:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.14, 2.40, 3.29 [00:31:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.01, 11.55, 11.93 [00:32:08] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.36, 5.67, 6.69 [00:32:58] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:33:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.85, 12.90, 12.36 [00:37:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.49, 7.90 [00:38:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.91, 3.60, 3.47 [00:40:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.93, 3.24, 3.37 [00:41:52] PROBLEM - mwtask141 Disk Space on mwtask141 is WARNING: DISK WARNING - free space: / 7704 MB (10% inode=90%); [00:43:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.45, 7.61, 7.73 [00:51:55] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.57, 7.21, 6.03 [00:54:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.70, 4.39, 3.43 [00:55:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.79, 7.81, 7.91 [00:57:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.48, 7.93, 7.93 [00:58:36] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.39, 3.65, 3.36 [00:59:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.17, 10.01, 11.63 [00:59:50] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.62, 6.17, 6.40 [01:00:35] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.96, 2.94, 3.13 [01:01:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.82, 7.90 [01:01:52] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 12592 MB (17% inode=90%); [01:03:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.43, 10.54, 11.43 [01:07:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.29, 10.25, 11.20 [01:09:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.40, 7.58 [01:15:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 7.59, 7.66 [01:16:37] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.33, 7.42, 6.60 [01:19:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.08, 8.34, 7.94 [01:21:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.59, 7.37, 7.64 [01:22:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 11.01, 11.89 [01:23:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.45, 7.94, 7.80 [01:24:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.80, 12.14, 12.20 [01:25:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.25, 11.36, 10.93 [01:27:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 7.96, 7.84 [01:27:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.81, 11.43, 11.00 [01:27:34] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1394547522 secs [01:29:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.35, 11.66, 11.13 [01:31:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.80, 11.63, 11.21 [01:31:34] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07714483142 secs [01:31:42] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:32:27] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.22, 7.76, 7.82 [01:33:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.29, 7.15, 7.41 [01:35:27] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.13, 3.76, 3.21 [01:35:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.97, 6.84, 7.25 [01:35:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.60, 11.56, 11.24 [01:37:27] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.28, 3.08, 3.02 [01:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.86, 11.41, 11.24 [01:38:24] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.09, 7.76, 7.80 [01:39:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.84, 6.00, 6.79 [01:43:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.15, 6.52, 6.87 [01:43:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.22, 11.95, 11.42 [01:44:20] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.92, 7.53, 7.97 [01:45:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.66, 6.05, 6.64 [01:47:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.80, 3.50, 3.11 [01:47:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.32, 11.73, 11.48 [01:49:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.10, 4.18, 3.40 [01:49:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.33, 12.93, 11.92 [01:51:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 7.19, 6.92 [01:52:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.83, 4.47, 6.42 [01:54:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.11, 3.84, 3.54 [01:57:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.56, 6.55, 6.78 [01:58:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.19, 3.94, 3.63 [02:00:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.46, 5.59, 2.92 [02:00:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.95, 3.66, 3.54 [02:03:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.59, 7.77, 7.21 [02:04:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.75, 2.92, 2.43 [02:05:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.55, 7.19 [02:06:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.84, 3.65, 3.53 [02:15:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.91, 6.21, 6.71 [02:16:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.64, 3.92, 3.91 [02:18:48] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [02:19:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.66, 7.47, 7.09 [02:20:46] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [02:21:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 7.47, 7.16 [02:23:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.44, 8.42, 7.53 [02:24:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.70, 2.54, 3.23 [02:25:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.71, 7.73, 7.39 [02:26:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.41, 3.70, 3.20 [02:27:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.95, 8.10, 7.57 [02:28:16] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.01, 2.71, 2.89 [02:29:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.68, 7.98, 7.59 [02:36:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.45, 3.77, 3.40 [02:37:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.62, 7.72, 7.52 [02:38:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.07, 4.03, 3.36 [02:38:24] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [02:38:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.73, 2.95, 3.14 [02:39:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.21, 7.30, 7.38 [02:40:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.06, 3.69, 3.32 [02:42:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.26, 3.93, 3.45 [02:44:12] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.63, 3.59, 3.40 [02:46:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.01, 2.98, 3.19 [02:46:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.98, 3.70, 3.44 [02:48:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.29, 2.94, 3.19 [02:49:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.52, 7.38, 7.32 [02:49:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.34, 6.96, 5.82 [02:51:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.15, 7.02, 7.21 [02:51:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.43, 7.42, 6.13 [02:55:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.41, 7.66, 7.39 [02:55:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.31, 7.22, 6.41 [02:57:08] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.72, 3.52, 3.34 [02:57:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.59, 7.69, 7.44 [02:57:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.00, 6.06, 6.08 [02:59:08] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.33, 3.21, 3.26 [02:59:52] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:01:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.76, 10.73, 11.98 [03:03:07] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.33, 4.60, 3.77 [03:03:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 10.96, 11.88 [03:05:06] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.59, 3.77, 3.56 [03:07:20] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:08:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.03, 3.44, 3.04 [03:09:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.09, 2.92, 3.26 [03:10:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.44, 2.63, 2.79 [03:11:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.73, 10.82, 11.70 [03:15:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.14, 7.10, 7.88 [03:15:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.82, 6.13, 6.65 [03:15:30] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.19, 6.71, 6.04 [03:17:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.46, 8.03, 8.12 [03:17:29] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.88, 5.71, 5.76 [03:19:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 6.95, 6.89 [03:19:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.83, 11.20, 11.40 [03:25:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 11.15, 11.44 [03:27:21] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.30, 6.61, 5.94 [03:27:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.98, 6.45, 6.76 [03:30:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.88, 3.69, 2.39 [03:32:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.72, 2.63, 2.16 [03:32:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.35, 4.03, 3.22 [03:33:18] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 2.80, 6.88, 6.70 [03:33:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.44, 10.76, 10.96 [03:35:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.92, 6.16, 6.47 [03:36:24] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.60, 11.28, 11.19 [03:39:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.59, 12.26, 11.56 [03:40:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.23, 7.12, 6.90 [03:42:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.60, 6.62, 6.75 [03:43:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.81, 11.95, 11.55 [03:45:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.67, 12.74, 11.87 [03:47:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.78, 7.84, 7.21 [03:48:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.22, 3.61, 3.92 [03:49:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.34, 7.31, 7.09 [03:52:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.93, 3.88, 3.95 [03:53:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.70, 8.04, 7.42 [03:54:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.21, 3.15, 3.67 [03:55:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.52, 7.89, 7.45 [03:55:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.81, 11.88, 11.99 [03:57:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.87, 13.95, 12.72 [03:58:36] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:01:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.63, 8.29, 7.63 [04:02:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.10, 2.58, 3.21 [04:05:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.07, 7.73, 7.57 [04:19:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.96, 10.19, 11.52 [04:23:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.72, 7.40, 7.97 [04:23:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.17, 7.37, 7.16 [04:24:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.74, 3.95, 3.29 [04:25:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.31, 7.75, 8.03 [04:25:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.67, 7.43, 7.21 [04:25:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.77, 11.73, 11.74 [04:26:47] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.25, 3.29, 3.13 [04:27:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.96, 7.54, 7.92 [04:27:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.27, 11.30, 11.57 [04:29:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.56, 8.08, 8.05 [04:31:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.20, 7.05 [04:31:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.28, 11.88, 11.63 [04:33:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 7.24, 7.08 [04:37:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.67, 6.05, 6.65 [04:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.58, 11.45, 11.68 [04:43:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.94, 11.58, 11.60 [04:45:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.53, 7.87, 7.99 [04:45:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.22, 10.82, 11.32 [04:46:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 10.90, 11.76 [04:50:28] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.08, 5.65, 3.52 [04:51:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.27, 7.82, 7.89 [04:52:27] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.57, 5.30, 3.66 [04:53:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.50, 7.41, 7.74 [04:53:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.28, 8.58, 10.03 [05:00:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.38, 9.11, 10.09 [05:03:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.21, 7.41, 7.45 [05:03:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.63, 10.35, 10.06 [05:05:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.28, 7.31, 7.40 [05:05:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.53, 9.71, 9.87 [05:07:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.34, 3.36, 2.77 [05:09:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.94, 2.82, 2.65 [05:09:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.50, 7.80, 7.55 [05:09:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.49, 10.79, 10.31 [05:11:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.83, 11.46, 10.49 [05:11:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.14, 10.07, 10.09 [05:13:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.95, 11.38, 10.59 [05:13:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.43, 7.84, 7.65 [05:16:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.45, 12.46, 11.19 [05:16:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.78, 4.16, 3.22 [05:17:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.61, 8.61, 7.95 [05:17:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.53, 11.40, 10.62 [05:18:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.95, 11.86, 11.15 [05:20:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.18, 3.42, 3.16 [05:20:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.25, 11.82, 11.20 [05:21:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.52, 7.95, 6.74 [05:22:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.77, 2.85, 2.98 [05:23:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.69, 7.98, 6.91 [05:24:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.10, 11.04, 11.06 [05:25:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.76, 11.52, 11.05 [05:26:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.03, 11.63, 11.26 [05:28:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.17, 11.37, 11.23 [05:30:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.56, 6.02, 3.03 [05:31:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.55, 6.53, 6.60 [05:31:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.36, 11.00, 10.85 [05:34:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.38, 3.47, 2.66 [05:34:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.24, 12.04, 11.48 [05:36:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.06, 2.66, 2.46 [05:36:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.94, 11.71, 11.44 [05:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.33, 11.86, 11.41 [05:38:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.46, 11.89, 11.51 [05:39:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.16, 12.67, 11.73 [05:41:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.72, 7.10, 6.79 [05:45:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.54, 7.96, 7.16 [05:47:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 7.53, 7.10 [05:48:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.60, 11.08, 11.59 [05:49:41] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [05:50:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.48, 12.23, 11.95 [05:51:44] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [05:55:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.45, 7.82, 7.33 [05:57:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.29, 6.94, 7.07 [06:00:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.85, 11.70, 11.88 [06:00:16] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.71, 4.22, 2.51 [06:01:54] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:02:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.94, 12.55, 12.17 [06:02:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.13, 3.62, 2.49 [06:03:51] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [06:04:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.60, 3.04, 2.42 [06:05:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.45, 6.31, 6.75 [06:17:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 7.35, 6.98 [06:23:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.87, 7.99, 7.43 [06:27:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.30, 7.26, 7.20 [06:29:00] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.81, 3.19, 2.23 [06:29:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 7.23, 7.20 [06:30:58] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.42, 2.49, 2.08 [06:31:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.13, 10.52, 11.89 [06:35:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.38, 11.86, 12.10 [06:36:34] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1189991832 secs [06:37:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.89, 6.87, 7.87 [06:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.73, 11.34, 11.86 [06:39:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.09, 6.31, 6.73 [06:41:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.84, 8.02, 8.08 [06:43:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.19, 7.73, 7.97 [06:44:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.88, 10.62, 11.69 [06:45:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.88, 8.26, 8.14 [06:46:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.07, 7.66, 7.12 [06:46:34] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08332371712 secs [06:47:50] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:48:00] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.99, 3.66, 2.30 [06:49:48] RECOVERY - cp22 APT on cp22 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [06:50:00] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.69, 2.88, 2.18 [06:51:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.90, 6.90, 7.01 [06:52:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.82, 10.81, 11.34 [06:58:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.30, 11.06, 11.50 [06:59:34] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.81, 6.11, 6.65 [07:05:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.71, 10.83, 10.57 [07:07:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.71, 7.54, 7.96 [07:07:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 10.60, 10.55 [07:08:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.07, 11.06, 11.19 [07:12:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.04, 11.80, 11.52 [07:15:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.96, 9.83, 10.20 [07:17:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.64, 7.22, 7.35 [07:18:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.74, 12.10, 11.61 [07:22:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.41, 11.23, 11.42 [07:23:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.10, 7.41, 7.48 [07:27:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.77, 7.44, 6.72 [07:29:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.44, 7.89, 7.62 [07:31:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.08, 7.64, 7.05 [07:33:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.28, 7.36, 7.51 [07:33:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.66, 6.49, 6.69 [07:35:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.23, 7.71, 7.62 [07:35:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.27, 9.62, 9.43 [07:36:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.06, 9.17, 10.11 [07:37:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.45, 7.62, 7.60 [07:37:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.65, 9.21, 9.31 [07:39:44] PROBLEM - mw121 APT on mw121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [07:41:45] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (1 critical updates). [07:42:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.66, 10.81, 10.49 [07:43:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.84, 7.74, 7.57 [07:45:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.48, 7.52, 7.04 [07:49:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.94, 7.69, 7.66 [07:52:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.51, 9.45, 10.06 [07:55:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.65, 7.83, 7.69 [07:55:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.33, 7.08 [07:57:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.49, 7.41, 7.56 [07:57:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.71, 6.91, 6.97 [07:59:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.07, 6.22, 6.71 [08:06:34] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1407476664 secs [08:13:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.43, 6.03, 6.77 [08:19:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.54, 7.32, 7.05 [08:21:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.72, 7.23, 7.06 [08:30:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:30:34] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07494565845 secs [08:39:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.50, 6.02, 6.64 [08:45:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.89, 6.94, 6.77 [08:49:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.18, 8.01, 7.21 [08:53:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 7.03, 6.52 [08:55:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.19, 7.87, 7.46 [08:55:34] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.95, 6.31, 6.32 [08:59:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.13, 7.69, 7.46 [09:05:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.63, 7.59, 7.58 [09:09:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.64, 6.92, 6.65 [09:11:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.89, 6.15, 6.40 [09:13:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.57, 7.52, 7.40 [09:33:42] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 18.13, 9.70, 4.81 [09:35:19] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [09:37:17] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [09:37:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 2.27, 6.90, 4.94 [09:39:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.31, 5.06, 4.50 [09:43:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 4.72, 6.84, 7.70 [09:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.41, 7.84, 7.87 [09:51:12] PROBLEM - db101 Backups SQL on db101 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209679 seconds old and 1879 bytes [09:53:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.53, 7.92, 7.95 [09:55:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.34, 8.51, 8.16 [09:58:38] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [09:59:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.03, 10.16, 8.83 [10:00:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.56, 2.84, 1.89 [10:01:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.96, 9.75, 8.84 [10:02:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.32, 2.24, 1.79 [10:08:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.56, 10.29, 9.41 [10:09:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.91, 7.75, 8.00 [10:12:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.63, 11.47, 10.08 [10:14:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.79, 11.21, 10.13 [10:20:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.18, 12.57, 10.95 [10:21:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.79, 8.84, 8.11 [10:22:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.97, 7.72, 6.94 [10:22:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.59, 12.10, 10.48 [10:26:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.34, 8.42, 7.36 [10:27:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 7.78, 7.26 [10:31:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.73, 6.00, 3.54 [10:33:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.18, 6.55, 4.07 [10:35:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.49, 8.00, 7.37 [10:35:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.67, 4.72, 3.70 [10:41:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.57, 7.99, 7.67 [10:47:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.45, 8.05, 7.68 [10:53:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.45, 7.53, 7.66 [10:57:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.15, 8.15, 7.84 [10:59:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 7.67, 7.70 [11:19:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.44, 7.61, 7.33 [11:23:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.59, 7.63, 7.42 [11:25:13] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.017 second response time [11:25:46] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:22] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.59, 2.81, 1.97 [11:27:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.22, 8.41, 7.74 [11:29:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 7.70, 7.56 [11:31:18] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.20, 2.56, 2.13 [11:35:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.16, 7.67, 7.41 [11:38:01] PROBLEM - itadminwiki.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'itadminwiki.pl' expires in 15 day(s) (Sun 25 Jun 2023 11:27:13 GMT +0000). [11:41:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.80, 7.81, 7.62 [11:45:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/feafa9b24781...d85c134df387 [11:45:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.06, 7.70, 7.59 [11:45:31] [02miraheze/ssl] 07MirahezeSSLBot 03d85c134 - Bot: Update SSL cert for itadminwiki.pl [11:47:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 7.87, 7.68 [11:49:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.93, 7.73 [11:58:41] Got this error in phab : #1021: Disk full (/tmp/#sql-temptable-c49-2a2fd3d-0.MAI); waiting for someone to free some space... (errno: 28 "No space left on device"). Disk space full ? [12:11:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.69, 7.36, 7.98 [12:15:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.29, 7.55, 7.90 [12:17:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.23, 7.35, 7.78 [12:21:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.30, 7.68, 7.77 [12:25:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.72, 7.96, 7.89 [12:35:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.52, 8.23, 7.91 [12:36:01] RECOVERY - itadminwiki.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'itadminwiki.pl' will expire on Thu 07 Sep 2023 10:45:19 GMT +0000. [12:41:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.64, 7.10, 7.95 [12:41:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.81, 7.86, 7.92 [12:43:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.95, 7.55, 7.99 [12:43:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.01, 8.32, 8.05 [12:45:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.30, 7.37, 7.87 [12:45:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.36, 7.60, 7.80 [12:47:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.45, 7.80, 7.97 [12:49:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.36, 8.53, 8.07 [12:53:55] PROBLEM - db112 MariaDB Connections on db112 is WARNING: Warning connection usage: 85.2%Current connections: 426 [12:57:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.27, 7.46, 7.84 [12:59:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.50, 8.33, 8.10 [13:01:19] paladox: do you exist? [13:01:41] I assume you haven't left and might be able to bandage phab [13:03:27] PROBLEM - db112 MariaDB Connections on db112 is CRITICAL: Critical connection usage: 90.4%Current connections: 452 [13:11:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.50, 7.83, 7.96 [13:13:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.50, 8.70, 8.28 [13:16:36] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.083 second response time [13:17:46] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [13:19:00] PROBLEM - db112 MariaDB on db112 is CRITICAL: Too many connections [13:23:56] @Orduin also ^^ [13:29:52] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:32:46] @orduin [13:43:01] [02ManageWiki] 07OAuthority opened pull request 03#410: fix log spam - 13https://github.com/miraheze/ManageWiki/pull/410 [13:43:11] I’m mobile [13:59:14] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:05:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.34, 7.16, 7.97 [14:09:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.20, 8.06, 8.12 [14:15:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.04, 7.26, 7.81 [14:19:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.97, 7.62, 7.78 [14:21:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.94, 7.87, 7.87 [14:23:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.68, 9.44, 8.44 [14:30:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.48, 3.37, 2.29 [14:32:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.25, 2.85, 2.22 [14:41:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 13.28, 8.55, 4.26 [14:43:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.89, 6.66, 4.09 [14:46:52] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1676 bytes in 0.055 second response time [14:46:59] PROBLEM - mon141 HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 335 bytes in 0.049 second response time [14:48:49] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 335 bytes in 0.088 second response time [14:48:55] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.080 second response time [14:50:45] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 40883 bytes in 0.376 second response time [14:56:41] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 335 bytes in 0.051 second response time [14:56:50] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 3.76, 7.12, 3.92 [14:58:37] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 40891 bytes in 0.061 second response time [15:00:40] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 0.60, 3.54, 3.15 [15:02:35] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.27, 2.48, 2.81 [15:13:51] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 2.20, 1.24, 0.72 [15:15:49] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 1.59, 1.44, 0.86 [15:31:03] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.97, 3.22, 2.23 [15:31:11] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:33:01] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.38, 2.51, 2.08 [15:33:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.45, 3.90, 2.88 [15:35:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.11, 3.15, 2.72 [15:43:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 6.89, 7.89 [15:45:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.05, 7.88, 8.14 [15:47:15] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 335 bytes in 0.060 second response time [15:48:39] PROBLEM - mon141 HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 335 bytes in 0.061 second response time [15:50:39] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.056 second response time [15:51:07] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 40891 bytes in 0.034 second response time [15:58:41] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 11.18, 5.09, 3.06 [16:00:33] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [16:02:37] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.93, 3.99, 3.15 [16:04:35] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.99, 2.89, 2.84 [16:06:44] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 335 bytes in 2.405 second response time [16:10:13] PROBLEM - db112 MariaDB Connections on db112 is UNKNOWN: [16:11:35] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:40] PROBLEM - nexus.wiki - reverse DNS on sslhost is UNKNOWN: [16:12:58] PROBLEM - mon141 HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 335 bytes in 2.907 second response time [16:13:27] PROBLEM - db112 MariaDB Connections on db112 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/1040): Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/1040): Too many connections in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Too many connections [16:13:42] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 335 bytes in 0.100 second response time [16:13:53] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 34.61, 27.37, 12.38 [16:15:01] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 3.02, 1.82, 1.01 [16:16:56] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 1.09, 1.66, 1.06 [16:31:17] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.19, 4.02, 2.88 [16:33:15] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.54, 3.10, 2.68 [16:33:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.51, 3.24, 2.81 [16:34:58] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 0.22, 0.87, 3.59 [16:35:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.84, 2.75, 2.68 [16:36:53] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.19, 0.66, 3.20 [16:41:33] PROBLEM - nexus.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nexus.wiki could not be found [16:45:51] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 1.82, 1.71, 1.07 [16:47:49] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.72, 1.29, 0.99 [16:51:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.67, 3.08, 2.72 [16:53:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.72, 2.55, 2.57 [16:59:16] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [17:07:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.21, 3.34, 2.72 [17:09:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.87, 2.67, 2.55 [17:15:54] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 5.15, 2.64, 1.30 [17:19:50] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 0.87, 1.97, 1.35 [17:21:50] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.25, 1.36, 1.20 [17:25:59] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.19, 0.69, 0.95 [17:27:39] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 2.96, 10.05, 5.94 [17:34:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 10.22, 11.75 [17:35:36] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 0.41, 2.31, 3.68 [17:37:36] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.61, 1.79, 3.32 [17:39:54] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.44, 4.08, 2.20 [17:40:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.02, 10.20, 11.18 [17:45:54] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.34, 3.17, 2.48 [17:45:55] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 3.08, 1.93, 1.16 [17:47:55] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 1.31, 1.82, 1.22 [17:47:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.78, 10.87, 11.34 [17:49:54] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.68, 1.51, 1.18 [17:49:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.29, 11.89, 11.63 [17:51:44] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:51:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.57, 11.98, 11.72 [17:52:04] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.061 second response time [17:53:04] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 20493 MB (15% inode=99%); [17:53:19] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 1213 bytes in 2.379 second response time [17:53:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.18, 12.36, 11.93 [17:56:04] PROBLEM - mon141 HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 335 bytes in 0.011 second response time [17:56:44] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:57:10] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 40883 bytes in 1.334 second response time [17:57:17] !log deleted mysql bin log files on db112 and restarted mysql [17:57:18] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20261 bytes in 0.305 second response time [17:57:18] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.634 second response time [17:57:23] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 20 Threads: 13 Questions: 13110 Slow queries: 282 Opens: 211 Open tables: 205 Queries per second avg: 655.500 [17:57:26] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.113 second response time [17:57:27] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 2.2%Current connections: 11 [17:57:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:04] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.040 second response time [17:59:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 18.80, 6.38, 3.14 [18:01:44] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:03:01] That's a dodgy alert [18:03:07] Thanks paladox [18:05:13] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.88, 3.30, 2.90 [18:06:44] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:08:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/27a6994eb81f [18:08:59] [02miraheze/puppet] 07paladox 0327a6994 - grafana: convert from mysql to sqlite [18:09:00] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [18:09:01] [02puppet] 07paladox opened pull request 03#3221: grafana: convert from mysql to sqlite - 13https://github.com/miraheze/puppet/pull/3221 [18:10:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/27a6994eb81f...9d6611ca2b73 [18:10:53] [02miraheze/puppet] 07paladox 039d6611c - Update db.pp [18:10:56] [02puppet] 07paladox synchronize pull request 03#3221: grafana: convert from mysql to sqlite - 13https://github.com/miraheze/puppet/pull/3221 [18:11:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/9d6611ca2b73...ebb3e5969693 [18:11:12] [02miraheze/puppet] 07paladox 03ebb3e59 - Delete grafana-grants.sql.erb [18:11:15] [02puppet] 07paladox synchronize pull request 03#3221: grafana: convert from mysql to sqlite - 13https://github.com/miraheze/puppet/pull/3221 [18:14:49] !log install sqlite3 temporarily on db112 [18:16:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:17:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.32, 7.47, 5.76 [18:19:06] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:19:30] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 15.20, 8.81, 6.36 [18:21:06] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [18:23:30] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 1.98, 6.87, 6.32 [18:23:40] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:25:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.27, 3.84, 2.94 [18:25:24] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.28, 5.03, 5.71 [18:27:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.72, 3.00 [18:28:21] [02puppet] 07paladox closed pull request 03#3221: grafana: convert from mysql to sqlite - 13https://github.com/miraheze/puppet/pull/3221 [18:28:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±2] 13https://github.com/miraheze/puppet/compare/17ede6f422a6...c9c653e7947f [18:28:25] [02miraheze/puppet] 07paladox 03c9c653e - grafana: convert from mysql to sqlite (#3221) [18:28:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [18:28:31] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [18:29:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.89, 2.99, 2.81 [18:36:54] !log mon141: upgrade grafana [18:37:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:39:22] [02ManageWiki] 07paladox reviewed pull request 03#410 commit - 13https://github.com/miraheze/ManageWiki/pull/410#discussion_r1224634222 [18:43:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.47, 6.46, 7.83 [18:49:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.93, 6.97, 5.79 [18:49:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.32, 7.46, 7.71 [18:50:55] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:51:13] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.90, 5.45, 5.36 [18:52:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0538ab32e621 [18:52:39] [02miraheze/puppet] 07paladox 030538ab3 - Support backing up Grafana db [18:52:41] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [18:52:47] [02puppet] 07paladox opened pull request 03#3222: Support backing up Grafana db - 13https://github.com/miraheze/puppet/pull/3222 [18:53:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0538ab32e621...c7d4d6f2c571 [18:53:46] [02miraheze/puppet] 07paladox 03c7d4d6f - Update init.pp [18:53:49] [02puppet] 07paladox synchronize pull request 03#3222: Support backing up Grafana db - 13https://github.com/miraheze/puppet/pull/3222 [18:57:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c7d4d6f2c571...a8a01e4fbbef [18:57:21] [02miraheze/puppet] 07paladox 03a8a01e4 - Update grafana.pp [18:57:23] [02puppet] 07paladox synchronize pull request 03#3222: Support backing up Grafana db - 13https://github.com/miraheze/puppet/pull/3222 [18:58:50] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.66, 7.85, 6.19 [19:00:30] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:01:26] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:03] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:29] PROBLEM - mon141 Backups Grafana on mon141 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/grafana-backup.log [19:03:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.039 second response time [19:03:43] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.91, 3.81, 2.61 [19:04:01] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.752 second response time [19:05:33] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.78, 3.26, 2.47 [19:07:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.51, 10.07, 11.98 [19:07:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 7.57, 7.91 [19:07:31] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.53, 3.71, 2.73 [19:07:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.13, 9.78, 11.77 [19:11:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.84, 7.75, 7.87 [19:11:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.41, 3.76, 3.01 [19:14:13] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.53, 7.24, 7.86 [19:15:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.04, 10.62, 11.27 [19:15:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.36, 7.82, 7.89 [19:15:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.96, 11.37, 11.77 [19:15:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.48, 3.12, 2.92 [19:15:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.18, 3.27, 3.35 [19:18:11] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.53, 7.47, 7.79 [19:19:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.99, 11.78, 11.62 [19:20:10] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.34, 7.04, 7.57 [19:20:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.81, 12.80, 12.01 [19:21:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.76, 7.92, 7.80 [19:22:08] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 17.75, 10.26, 8.63 [19:23:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.37, 3.41, 3.07 [19:23:45] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.36, 3.56, 3.31 [19:25:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.39, 7.33, 7.60 [19:25:12] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.25, 3.31, 3.08 [19:25:44] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.83, 3.52, 3.32 [19:28:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.43, 11.31, 11.70 [19:29:06] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [19:29:41] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.71, 2.96, 3.18 [19:30:04] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.64, 6.93, 7.80 [19:31:34] [02puppet] 07paladox synchronize pull request 03#3222: Support backing up Grafana db - 13https://github.com/miraheze/puppet/pull/3222 [19:31:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a8a01e4fbbef...b82025bd1afa [19:31:38] [02miraheze/puppet] 07paladox 03b82025b - Update miraheze-backup.py.erb [19:32:02] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.44, 8.20, 8.15 [19:32:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.16, 11.61, 11.71 [19:34:01] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.05, 7.59, 7.96 [19:34:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.11, 7.60, 7.38 [19:35:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.54, 11.21, 11.93 [19:36:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 7.86, 8.25, 8.19 [19:37:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.25, 11.49, 11.93 [19:38:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.38, 7.26, 7.32 [19:41:48] [02puppet] 07paladox synchronize pull request 03#3222: Support backing up Grafana db - 13https://github.com/miraheze/puppet/pull/3222 [19:41:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b82025bd1afa...3db97d20b934 [19:41:52] [02miraheze/puppet] 07paladox 033db97d2 - Update miraheze-backup.py.erb [19:42:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.06, 7.85, 7.52 [19:42:54] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.56, 3.21 [19:45:54] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.18, 7.10, 7.90 [19:46:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.86, 3.39, 3.23 [19:48:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.73, 7.92, 7.72 [19:51:50] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.53, 4.80, 6.56 [19:54:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 7.78, 7.65 [19:56:28] [02puppet] 07dependabot[bot] created branch 03dependabot/bundler/modules/graylog/puppet-6.25.1 - 13https://github.com/miraheze/puppet [19:56:29] [02miraheze/puppet] 07dependabot[bot] pushed 031 commit to 03dependabot/bundler/modules/graylog/puppet-6.25.1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/550826a65ddb [19:56:32] [02miraheze/puppet] 07dependabot[bot] 03550826a - build(deps): bump puppet from 4.4.2 to 6.25.1 in /modules/graylog [19:56:33] [02puppet] 07dependabot[bot] labeled pull request 03#3223: build(deps): bump puppet from 4.4.2 to 6.25.1 in /modules/graylog - 13https://github.com/miraheze/puppet/pull/3223 [19:56:35] [02puppet] 07dependabot[bot] opened pull request 03#3223: build(deps): bump puppet from 4.4.2 to 6.25.1 in /modules/graylog - 13https://github.com/miraheze/puppet/pull/3223 [19:56:38] [02puppet] 07dependabot[bot] labeled pull request 03#3223: build(deps): bump puppet from 4.4.2 to 6.25.1 in /modules/graylog - 13https://github.com/miraheze/puppet/pull/3223 [19:58:38] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:02:13] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [20:02:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.84, 3.48, 2.86 [20:02:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.18, 7.29, 7.56 [20:04:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.65, 3.85, 3.07 [20:04:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.66, 7.62, 7.63 [20:06:15] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.89, 3.45, 3.01 [20:10:04] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.79, 2.85, 2.87 [20:12:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 7.71, 7.81 [20:14:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.93, 8.36, 8.02 [20:18:23] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.4587317705 secs [20:22:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.51, 7.75, 7.91 [20:24:20] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -5.701184273e-05 secs [20:28:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.31, 8.86, 8.24 [20:28:38] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:31:43] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:34:23] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.40, 6.70, 4.62 [20:34:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.97, 7.83, 7.98 [20:36:22] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.87, 4.87, 4.21 [20:38:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.43, 7.93, 7.98 [20:40:09] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.105907619 secs [20:40:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.20, 6.94, 7.61 [20:42:08] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02456018329 secs [20:47:57] [02puppet] 07paladox closed pull request 03#3222: Support backing up Grafana db - 13https://github.com/miraheze/puppet/pull/3222 [20:48:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/c9c653e7947f...5251e226d507 [20:48:01] [02miraheze/puppet] 07paladox 035251e22 - Support backing up Grafana db (#3222) [20:48:02] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [20:48:04] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [20:57:25] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:58:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.44, 7.78, 7.35 [20:59:30] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.04, 4.26, 3.32 [21:01:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.21, 3.74, 3.24 [21:02:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 7.63, 7.41 [21:03:12] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:03:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.90, 4.26, 3.47 [21:04:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.80, 7.50 [21:06:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.59, 7.76, 7.53 [21:08:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.08, 8.07, 7.68 [21:11:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.23, 3.85, 3.71 [21:12:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.01, 7.33, 7.48 [21:15:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.25, 3.07, 3.39 [21:18:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.50, 8.21, 7.84 [21:20:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.39, 7.61, 7.67 [21:24:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.49, 8.83, 8.05 [21:30:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.87, 6.89, 7.52 [21:32:42] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:38:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.06, 7.24, 7.35 [21:40:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 7.35, 7.37 [21:46:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.11, 7.53, 7.44 [21:47:06] !log mon141: grafana-cli plugins install grafana-opensearch-datasource [21:47:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:19] !log mon141: switch to opensearch plugin for os data [21:47:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:55] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.96, 4.36, 3.04 [21:53:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.85, 3.83, 3.00 [21:55:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.12, 3.21, 2.87 [21:58:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.59, 7.30, 7.74 [22:00:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.27, 7.64, 7.81 [22:00:29] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.64, 5.73, 3.38 [22:02:27] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.70, 4.06, 3.05 [22:10:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5251e226d507...2d894207285d [22:10:51] [02miraheze/puppet] 07paladox 032d89420 - prometheus::es_exporter: elasticsearch.m.org -> opensearch.m.org [22:12:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.41, 3.28, 2.83 [22:14:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.67, 3.03, 2.80 [22:17:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.53, 3.57, 2.91 [22:19:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.86, 2.82, 2.71 [22:24:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.32, 7.96 [22:26:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.65, 8.27, 8.24 [22:30:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.57, 7.08, 7.99 [22:32:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.22, 7.24, 7.82 [22:34:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.59, 7.45, 7.89 [22:38:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.85, 4.04, 3.16 [22:39:53] !log [paladox@mwtask141] starting deploy of {'world': True} to mwtask141 [22:39:57] !log [paladox@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [22:40:02] !log [paladox@mwtask141] starting deploy of {'config': True} to mwtask141 [22:40:03] !log [paladox@mwtask141] finished deploy of {'config': True} to mwtask141 - SUCCESS in 0s [22:40:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:40:18] !log [@mwtask141] starting deploy of {'config': True} to all [22:40:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:40:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:40:27] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [22:40:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:40:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:40:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:44:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.85, 7.10, 7.41 [22:46:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.14, 7.01, 7.33 [22:46:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.38, 3.99, 3.61 [22:46:45] !log [paladox@mwtask141] starting deploy of {'config': True} to mwtask141 [22:46:46] !log [paladox@mwtask141] finished deploy of {'config': True} to mwtask141 - SUCCESS in 0s [22:46:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:08] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/paladox/oojsplus.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable oojsplus [22:47:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:38] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:50:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.71, 8.18, 7.74 [22:50:39] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.43, 3.06, 3.31 [22:51:40] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [22:52:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.05, 7.47, 7.54 [22:58:33] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php [22:58:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.30, 8.31, 7.65 [23:02:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.35, 7.94, 7.60 [23:08:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.33, 7.37 [23:10:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.84, 7.40, 7.39 [23:14:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.81, 10.51, 11.78 [23:18:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.06, 7.24 [23:24:41] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.84, 4.52, 3.16 [23:26:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.18, 11.48, 11.64 [23:26:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.17, 7.40, 7.76 [23:28:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.84, 7.88 [23:30:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.03, 7.41, 7.71 [23:31:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.30, 5.97, 3.69 [23:31:35] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:32:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.74, 11.91, 11.91 [23:34:14] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [23:34:15] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.45, 3.71, 3.42 [23:35:30] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.45, 6.96, 4.69 [23:36:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.27, 2.90, 3.16 [23:36:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.46, 13.56, 12.45 [23:37:28] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.45, 4.91, 4.21 [23:38:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.57, 7.13, 7.42 [23:43:55] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.43, 3.94, 3.43 [23:45:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.12, 3.70, 3.41 [23:47:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.03, 2.95, 3.16