[00:00:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:01:29] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.49, 3.21, 2.64 [00:02:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.99, 6.10, 3.27 [00:02:26] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 1.77, 2.75, 3.25 [00:02:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 27.44, 12.06, 5.94 [00:03:03] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5936 bytes in 0.012 second response time [00:03:23] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.56, 6.76, 4.51 [00:03:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 12.43, 10.86, 6.01 [00:04:59] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.579 second response time [00:05:11] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.847 second response time [00:05:29] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 2 backends are down. mw121 mw141 [00:05:47] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw132 [00:05:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.95, 3.37, 3.02 [00:06:19] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw142 [00:06:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.35, 3.44, 3.45 [00:06:50] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw141 [00:07:29] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [00:07:46] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [00:08:14] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.52, 3.75, 3.10 [00:08:19] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [00:08:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.10, 3.75, 3.57 [00:08:50] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [00:09:52] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.524 second response time [00:09:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.95, 4.04, 3.34 [00:10:14] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.98, 2.80, 2.83 [00:11:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 6.74, 4.48, 3.67 [00:14:42] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 10.76, 8.64, 4.51 [00:16:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.25, 3.98 [00:17:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.81, 3.10, 3.73 [00:20:39] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.63, 3.76, 3.72 [00:21:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.00, 3.81, 3.76 [00:22:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.42, 4.00 [00:22:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 2.41, 3.36 [00:23:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.95, 2.83, 3.36 [00:24:37] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.02, 2.40, 3.17 [00:25:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.53, 3.85, 3.76 [00:27:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.30, 3.06, 3.96 [00:27:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.33, 3.66, 3.70 [00:29:10] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.84, 3.60, 3.88 [00:30:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.52, 3.98, 3.84 [00:32:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.53, 2.49, 3.84 [00:32:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.82, 3.80 [00:33:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.07, 3.81, 3.72 [00:34:46] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 1.76, 2.45, 3.37 [00:35:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.31, 3.79, 3.73 [00:36:00] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.01, 3.18, 3.79 [00:36:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.43, 4.33, 3.98 [00:37:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.07, 3.71, 3.69 [00:38:00] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.71, 2.84, 3.58 [00:38:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.74, 3.82 [00:39:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.45, 3.79, 3.74 [00:39:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.42, 2.46, 3.37 [00:41:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.62, 4.19, 3.90 [00:42:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.05, 2.73, 3.37 [00:44:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.49, 3.71, 3.65 [00:45:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.19, 3.90, 3.88 [00:49:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.86, 4.02, 3.91 [00:50:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.93, 2.85, 3.99 [00:51:41] [02ssl] 07AgentIsai opened pull request 03#614: Remove redirects for wikis which don't point to Miraheze/have been permanently deleted - 13https://github.com/miraheze/ssl/pull/614 [00:51:42] [url] Page not found · GitHub · GitHub | github.com [00:51:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.63, 3.75, 3.82 [00:54:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.34, 4.24, 4.18 [00:54:15] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.47, 3.69, 3.47 [00:58:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.36, 3.23, 3.79 [00:58:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.41, 10.37, 9.44 [00:58:21] [02ssl] 07AgentIsai opened pull request 03#615: Remove certs for wikis which no longer point to us - 13https://github.com/miraheze/ssl/pull/615 [00:58:22] [url] Page not found · GitHub · GitHub | github.com [01:00:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.28, 3.73, 3.55 [01:00:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.59, 9.77, 9.34 [01:00:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.00, 9.87, 8.84 [01:01:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.27, 3.48, 3.47 [01:02:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.67, 5.56, 4.60 [01:02:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.43, 8.40, 8.44 [01:03:17] [02ssl] 07AgentIsai synchronize pull request 03#615: Remove certs for wikis which no longer point to us - 13https://github.com/miraheze/ssl/pull/615 [01:03:18] [url] Page not found · GitHub · GitHub | github.com [01:03:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.80, 3.44 [01:05:48] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.08, 3.28, 3.39 [01:05:49] [02ssl] 07AgentIsai synchronize pull request 03#615: Remove certs for wikis which no longer point to us - 13https://github.com/miraheze/ssl/pull/615 [01:05:50] [url] Page not found · GitHub · GitHub | github.com [01:05:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 3.37, 3.33 [01:05:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.02, 3.49, 3.53 [01:06:14] [02ssl] 07AgentIsai synchronize pull request 03#615: Remove certs for wikis which no longer point to us - 13https://github.com/miraheze/ssl/pull/615 [01:06:15] [url] Page not found · GitHub · GitHub | github.com [01:06:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.22, 3.54, 3.95 [01:06:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.74, 6.62, 4.27 [01:07:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.23, 3.38, 3.49 [01:09:40] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.87, 3.53, 3.45 [01:09:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.56, 3.46 [01:09:52] [02ssl] 07AgentIsai synchronize pull request 03#615: Remove certs for wikis which no longer point to us - 13https://github.com/miraheze/ssl/pull/615 [01:09:52] [url] Page not found · GitHub · GitHub | github.com [01:09:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.72, 3.93, 3.68 [01:10:17] [02ssl] 07AgentIsai synchronize pull request 03#615: Remove certs for wikis which no longer point to us - 13https://github.com/miraheze/ssl/pull/615 [01:10:17] ... [01:10:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.06, 3.55, 3.84 [01:11:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.79, 3.22, 3.34 [01:11:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.15, 3.67, 3.62 [01:12:44] PROBLEM - wiki.slimeapedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.slimeapedia.org All nameservers failed to answer the query. [01:12:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.11, 3.30, 3.58 [01:13:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.32, 3.65, 3.50 [01:13:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.34, 3.92, 3.72 [01:14:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.74, 3.90, 3.75 [01:15:28] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.28, 3.68, 3.54 [01:15:42] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.10, 3.73, 3.16 [01:16:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.92, 3.69, 3.70 [01:17:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.60, 3.46 [01:18:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.10, 3.23, 3.91 [01:19:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.74, 2.95, 3.23 [01:19:19] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.77, 4.16, 3.77 [01:19:36] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.68, 3.70, 3.29 [01:20:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.29, 2.78 [01:20:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.35, 2.97, 3.40 [01:21:32] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.86, 3.29, 3.18 [01:22:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 2.86, 2.68 [01:23:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.64, 3.61, 3.64 [01:24:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.71, 4.19, 4.00 [01:24:41] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 10.77, 5.82, 3.74 [01:25:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 5.99, 4.39, 3.91 [01:25:28] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.58, 3.79, 3.38 [01:25:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.73, 3.66, 3.92 [01:26:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.68, 3.94, 3.93 [01:27:25] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.88, 3.39, 3.28 [01:28:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.53, 3.74, 3.78 [01:30:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.43, 4.95, 4.26 [01:31:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.60, 3.34 [01:33:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 3.20, 3.22 [01:34:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.73, 3.74, 3.93 [01:34:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.95, 3.54, 3.61 [01:35:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.09, 4.05, 3.79 [01:36:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.64, 3.20, 3.92 [01:36:39] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.28, 3.47, 3.58 [01:37:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.89, 4.13, 3.45 [01:37:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.31, 3.40, 3.58 [01:38:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.29, 4.12, 4.03 [01:38:23] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 12.28, 5.70, 4.68 [01:38:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.26, 3.55, 3.98 [01:40:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.15, 3.92, 3.98 [01:40:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 3.73, 3.81, 4.03 [01:40:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.08, 3.69, 3.63 [01:41:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.68, 3.23, 3.27 [01:41:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.54, 2.96, 3.37 [01:42:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.88, 4.79, 4.32 [01:42:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.83, 3.79, 3.99 [01:42:40] PROBLEM - wiki.slimeapedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.slimeapedia.org could not be found [01:43:03] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.03, 3.65, 3.39 [01:44:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.96, 3.77, 4.00 [01:44:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.56, 3.11, 3.84 [01:44:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 3.96, 4.03, 4.06 [01:44:59] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.89, 3.27, 3.27 [01:45:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 10.41, 9.21 [01:46:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.32, 3.84, 4.00 [01:47:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.46, 9.29, 8.94 [01:47:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.48, 3.42, 3.24 [01:49:45] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.29, 3.24, 3.19 [01:50:02] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.44, 2.34, 3.29 [01:50:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.39, 3.96, 3.97 [01:51:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.57, 3.96, 3.63 [01:52:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.77, 3.48, 3.79 [01:52:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.46, 3.54, 3.73 [01:52:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.64, 3.42 [01:54:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.63, 3.00, 3.22 [01:55:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.56, 3.97, 3.72 [01:56:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.64, 3.61, 3.99 [01:56:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.93, 3.99, 3.86 [01:56:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.49, 3.39, 3.57 [01:57:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.95, 4.52, 3.97 [01:58:03] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.14, 2.81, 3.32 [01:58:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.67, 3.17 [01:58:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.06, 3.73, 3.99 [02:00:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.21, 3.06 [02:00:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.48, 3.75, 3.97 [02:02:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.31, 4.39, 4.18 [02:04:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.44, 3.93, 3.97 [02:06:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.16, 3.92, 3.98 [02:07:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.55, 3.61, 3.84 [02:08:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.42, 3.56, 3.92 [02:09:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.87, 3.45, 3.19 [02:10:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.31, 3.98, 3.93 [02:11:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.81, 3.26, 3.15 [02:11:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.06, 3.72, 3.83 [02:12:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.61, 3.44, 3.74 [02:13:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.68, 3.34, 3.67 [02:14:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.05, 3.70, 3.42 [02:14:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.55, 3.88, 3.84 [02:16:26] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.18, 4.56, 3.77 [02:16:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.14, 2.68, 3.29 [02:17:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 1.75, 2.70, 3.35 [02:18:22] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.19, 3.71, 3.55 [02:18:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.61, 3.89, 3.90 [02:19:38] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.78, 3.83, 3.39 [02:20:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.33, 3.43, 3.44 [02:20:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.17, 3.90, 3.89 [02:21:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.12, 3.21, 3.23 [02:22:16] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.10, 2.99, 3.29 [02:22:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.30, 3.88, 3.90 [02:22:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.54, 3.10, 3.32 [02:23:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.76, 3.34, 3.43 [02:25:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.55, 3.66, 3.53 [02:28:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.86, 3.61, 2.39 [02:28:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.98, 4.34, 4.02 [02:29:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.11, 3.14, 3.38 [02:30:14] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.80, 3.00, 2.32 [02:32:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.44, 3.62, 3.33 [02:33:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.45, 4.02, 3.66 [02:34:16] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.22, 3.23, 2.87 [02:34:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.99, 3.47, 3.31 [02:35:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.88, 3.68, 3.59 [02:36:13] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.38, 3.27, 2.93 [02:36:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.28, 3.73, 3.92 [02:36:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.70, 3.27, 3.26 [02:41:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.95, 3.19, 3.38 [02:43:32] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.58, 3.46, 2.90 [02:44:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.47, 3.89, 3.71 [02:44:36] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.79, 5.13, 3.51 [02:45:30] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.79, 2.89, 2.76 [02:46:18] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 11.06, 6.93, 3.64 [02:46:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.75, 4.34, 3.12 [02:48:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.71, 3.76, 3.70 [02:48:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.82, 3.33, 2.90 [02:48:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.40, 4.22, 3.91 [02:49:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.04, 3.26, 2.96 [02:50:16] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.95, 3.49, 2.96 [02:50:26] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.29, 3.96, 3.50 [02:50:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.33, 3.84, 3.81 [02:52:15] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.58, 2.51, 2.66 [02:54:19] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.66, 3.38, 3.39 [02:58:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.12, 3.65, 3.41 [03:00:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.10, 4.14, 3.92 [03:00:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.76, 3.72, 3.46 [03:02:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.77, 3.49, 3.70 [03:02:41] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.32, 3.16, 3.28 [03:04:26] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.64, 2.99, 3.32 [03:05:59] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.21, 3.28, 3.16 [03:06:37] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.24, 3.46, 3.38 [03:08:35] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.91, 3.32, 3.34 [03:09:49] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.83, 3.16, 3.15 [03:12:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.87, 3.91, 3.58 [03:15:02] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.07, 3.51, 2.55 [03:16:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.78, 3.78, 3.65 [03:18:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.69, 3.17, 3.40 [03:21:00] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.40, 3.43, 2.87 [03:21:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.03, 1.69, 3.72 [03:22:58] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.08, 2.56, 2.62 [03:23:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.11, 4.46, 4.48 [03:24:26] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.50, 3.10, 3.39 [03:29:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.26, 2.96, 3.87 [03:33:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.24, 2.19, 3.35 [03:39:24] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.64, 3.78, 3.34 [03:41:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.62, 2.95, 3.09 [03:47:15] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.04, 3.67, 3.25 [03:48:30] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.51, 3.38, 2.66 [03:49:11] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.76, 3.56, 3.25 [03:50:27] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.34, 3.35, 2.74 [03:51:05] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.27, 3.01, 3.08 [03:51:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.69, 2.90 [03:54:23] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.54, 3.58, 2.96 [03:55:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.82, 3.15, 2.90 [03:56:20] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.65, 3.22, 2.90 [04:02:04] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.39, 4.03, 3.24 [04:04:03] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.60, 3.12, 3.00 [04:04:50] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.51, 3.24, 2.93 [04:06:49] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.98, 3.03, 2.89 [04:14:37] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.99, 3.30, 3.04 [04:16:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.86, 3.77, 3.24 [04:18:29] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.18, 3.45, 3.18 [04:22:26] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.21, 3.12, 3.14 [04:22:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.28, 3.31, 2.99 [04:23:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 3.33, 2.59 [04:24:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.99, 3.37, 3.04 [04:25:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.25, 2.99, 2.56 [04:26:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.10, 2.85, 2.89 [04:28:50] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.19, 4.14, 3.16 [04:29:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.37, 2.66 [04:30:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.46, 2.77 [04:32:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.85, 4.21, 3.12 [04:34:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.82, 3.09 [04:36:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.77, 3.14 [04:38:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.57, 3.14 [04:38:45] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.82, 3.40, 3.01 [04:40:30] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.18, 3.66, 3.80 [04:40:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.49, 3.22, 3.06 [04:40:39] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.ultravanilla.world.key] [04:40:45] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.54, 3.02, 2.92 [04:44:24] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.83, 3.63, 3.72 [04:46:20] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.51, 2.79, 3.40 [04:54:26] PROBLEM - dragonquestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dragonquestwiki.com' expires in 15 day(s) (Mon 28 Nov 2022 04:44:57 GMT +0000). [04:55:11] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 11.14, 5.15, 3.40 [05:00:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f9ea6cd72430...925898e9e79c [05:00:47] [url] Comparing f9ea6cd72430...925898e9e79c · miraheze/ssl · GitHub | github.com [05:00:49] [02miraheze/ssl] 07MirahezeSSLBot 03925898e - Bot: Update SSL cert for dragonquestwiki.com [05:05:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.50, 3.26, 3.59 [05:08:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.30, 3.47, 2.93 [05:08:43] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:09:59] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.27, 2.97, 2.80 [05:11:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.07, 2.55, 3.21 [05:16:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.49, 10.86, 8.65 [05:16:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.59, 2.77, 2.03 [05:16:55] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.71, 2.74, 1.80 [05:18:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.29, 8.94, 8.20 [05:18:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.22, 2.21, 1.91 [05:18:22] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.05, 3.76, 3.01 [05:18:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.20, 3.69, 2.74 [05:18:53] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.00, 2.05, 1.67 [05:20:22] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:45] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.40, 3.89, 2.83 [05:21:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.22, 3.44, 2.95 [05:22:45] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.27, 3.72, 2.90 [05:23:22] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.12, 3.33, 2.97 [05:24:27] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.374 second response time [05:24:45] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.70, 3.26, 2.82 [05:26:09] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.08, 3.62, 3.42 [05:26:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.33, 3.23, 3.18 [05:26:32] PROBLEM - crocwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'crocwiki.com' expires in 15 day(s) (Mon 28 Nov 2022 05:01:56 GMT +0000). [05:28:05] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.77, 2.94, 3.18 [05:30:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.72, 3.40 [05:32:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/925898e9e79c...7cc377a1cbbd [05:32:03] [url] Comparing 925898e9e79c...7cc377a1cbbd · miraheze/ssl · GitHub | github.com [05:32:03] [02miraheze/ssl] 07MirahezeSSLBot 037cc377a - Bot: Update SSL cert for crocwiki.com [05:32:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 3.39, 3.33 [05:33:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.56, 3.51, 3.19 [05:35:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.48, 3.18, 3.11 [05:38:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.85, 3.53, 2.60 [05:42:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.85, 10.12, 8.73 [05:42:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 3.09, 2.67 [05:43:43] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.65, 3.45, 3.15 [05:44:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.85, 8.51, 8.31 [05:44:55] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.01, 3.70, 3.25 [05:45:39] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.03, 2.91, 2.99 [05:46:49] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.62, 3.72, 3.31 [05:48:44] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.40, 3.26, 3.19 [05:53:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.73, 3.63, 3.33 [05:53:59] RECOVERY - dragonquestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dragonquestwiki.com' will expire on Fri 10 Feb 2023 04:00:40 GMT +0000. [05:57:18] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.46, 3.17, 3.21 [06:00:40] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.11, 3.48, 2.86 [06:02:37] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.97, 2.79, 2.68 [06:11:26] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.02, 3.77, 3.30 [06:13:25] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.28, 3.41, 3.22 [06:13:46] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.21, 3.48, 3.19 [06:14:21] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.81, 2.71, 1.88 [06:15:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.23, 3.14, 2.85 [06:16:20] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.89, 2.89, 2.06 [06:16:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.82, 4.35, 3.45 [06:17:23] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.21, 3.37, 3.24 [06:17:49] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.83, 3.11, 2.88 [06:19:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.85, 3.85, 3.54 [06:20:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.45, 3.58, 3.37 [06:22:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.22, 4.42, 3.30 [06:22:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.51, 3.39, 3.33 [06:23:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.18, 3.74, 3.51 [06:24:35] RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Fri 10 Feb 2023 04:31:56 GMT +0000. [06:25:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.87, 3.65, 3.50 [06:27:17] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 1.98, 3.10, 3.32 [06:28:13] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 3.12, 4.32, 2.85 [06:28:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.53, 3.28, 1.83 [06:30:12] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.43, 3.71, 2.81 [06:30:14] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.44, 2.63, 1.78 [06:32:11] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.76, 3.32, 2.79 [06:32:42] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.10, 3.65, 2.94 [06:33:49] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.75, 3.15, 3.62 [06:34:38] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.02, 3.25, 2.88 [06:35:45] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.81, 2.40, 3.29 [06:38:29] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.48, 3.78, 3.20 [06:40:25] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.58, 4.11, 3.39 [06:46:03] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.82, 3.06, 2.50 [06:46:49] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.84, 3.41, 3.07 [06:48:12] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.08, 3.56, 3.52 [06:48:44] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.02, 3.40, 3.12 [06:52:00] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.67, 3.29, 2.96 [06:52:06] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.28, 3.10, 3.35 [06:52:53] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.01, 3.41, 2.70 [06:54:49] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.69, 3.18, 2.71 [06:55:04] PROBLEM - wiki.auraxis.co - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.auraxis.co. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:55:15] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.24, 3.56, 2.98 [06:56:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.01, 3.53, 3.52 [06:57:12] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.27, 3.69, 3.10 [07:00:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.56, 4.92, 4.04 [07:01:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.75, 3.05, 3.00 [07:01:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.92, 3.26 [07:03:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.56, 3.21 [07:04:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.47, 3.85, 3.81 [07:04:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.63, 3.10, 2.68 [07:04:45] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.27, 3.64, 3.02 [07:05:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.52, 3.22 [07:06:42] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.88, 3.01, 2.70 [07:06:45] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.17, 3.50, 3.05 [07:07:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.57, 3.28 [07:08:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.55, 3.84, 3.77 [07:08:45] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.54, 3.22, 3.01 [07:09:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 3.06, 3.13 [07:10:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.34, 3.62, 3.70 [07:14:03] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.94, 2.59, 3.25 [07:22:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.64, 4.02, 3.20 [07:22:45] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.85, 3.46, 3.08 [07:24:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.58, 3.64, 3.24 [07:24:08] RECOVERY - wiki.auraxis.co - reverse DNS on sslhost is OK: SSL OK - wiki.auraxis.co reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:24:45] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.53, 3.85, 3.27 [07:26:30] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:37] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.035 second response time [07:28:45] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.85, 3.33, 3.27 [07:30:20] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:31:14] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 4 backends are down. mw121 mw122 mw132 mw141 [07:31:46] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 3 backends are down. mw122 mw132 mw142 [07:32:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.740 second response time [07:32:52] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:54] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.540 second response time [07:35:18] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [07:35:46] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [07:37:43] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.22, 4.52, 3.74 [07:42:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:43:33] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 1.71, 3.89, 3.80 [07:44:09] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:27] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.31, 2.55, 3.27 [07:48:14] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.290 second response time [07:49:21] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw122 [07:49:46] PROBLEM - vise.dayid.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.40245246887207 seconds [07:49:46] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw122 [07:50:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.42, 3.25, 2.61 [07:50:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.552 second response time [07:51:20] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [07:51:22] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.53, 3.43, 3.50 [07:51:46] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [07:52:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 1.64, 2.59, 2.44 [07:55:16] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.20, 3.38, 3.45 [07:55:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.81, 3.51, 2.90 [07:57:12] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.90, 3.30, 3.42 [07:57:22] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.98, 3.24, 2.87 [07:59:09] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.70, 2.89, 3.25 [08:01:16] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.67, 3.66, 3.14 [08:03:12] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.51, 3.19, 3.02 [08:06:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.55, 3.05, 2.04 [08:08:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.96, 2.47, 3.92 [08:08:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.34, 2.44, 1.94 [08:10:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.14, 1.83, 3.83 [08:14:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.58, 3.71, 3.83 [08:14:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 2.95, 3.76 [08:16:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.15, 3.20, 3.62 [08:16:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 2.71, 3.57 [08:18:03] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.77, 2.68, 3.38 [08:19:10] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:22:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.33, 2.39, 3.20 [08:22:59] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.97, 3.49, 3.12 [08:24:57] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 3.15, 3.39, 3.12 [08:25:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.65, 3.53, 2.35 [08:27:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.45, 3.24, 2.39 [08:28:50] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.83, 3.86, 3.39 [08:30:47] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.48, 3.33, 3.25 [08:36:38] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.58, 3.50, 2.53 [08:38:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.60, 3.27, 2.58 [08:38:40] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.90, 3.94, 3.56 [08:46:40] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.84, 3.64, 3.51 [08:48:40] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.60, 3.21, 3.37 [08:55:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.06, 4.93, 3.26 [08:56:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.97, 6.12, 3.89 [08:56:45] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:32] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.95, 3.65, 2.89 [09:01:29] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.65, 4.01, 3.09 [09:04:19] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 2 backends are down. mw122 mw132 [09:05:05] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.280 second response time [09:06:19] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [09:07:28] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.41, 3.41, 3.23 [09:08:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.95, 3.22, 2.61 [09:09:24] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.70, 3.32, 3.22 [09:10:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.09, 2.36, 2.37 [09:11:17] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw122 [09:13:16] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [09:22:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.02, 2.18, 3.85 [09:22:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.19, 2.14, 3.89 [09:26:04] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.60, 1.80, 3.27 [09:26:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.33, 1.89, 3.35 [09:29:32] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2763153315 secs [09:29:58] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.94, 5.05, 3.97 [09:33:51] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.08, 3.99, 3.78 [09:35:47] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.17, 4.18, 3.86 [09:37:32] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02793419361 secs [09:37:44] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.51, 3.64, 3.71 [09:41:37] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.60, 2.81, 3.36 [09:57:06] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 12.33, 7.11, 3.86 [10:02:10] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 14.41, 10.56, 5.58 [10:02:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.80, 8.82, 5.08 [10:04:32] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw132 [10:04:50] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw121 [10:05:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.73, 3.24, 3.43 [10:05:11] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw141 [10:06:30] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [10:06:50] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [10:07:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.79, 2.42, 3.10 [10:07:11] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [10:10:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.65, 3.24, 3.89 [10:10:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.59, 3.34, 3.80 [10:14:03] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.17, 2.10, 3.27 [10:14:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.84, 2.32, 3.26 [10:27:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.30, 3.77, 2.81 [10:30:14] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.51, 3.85, 2.37 [10:30:45] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.43, 3.51, 2.89 [10:31:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.45, 2.78, 2.64 [10:32:14] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.56, 2.94, 2.21 [10:32:45] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.06, 3.11, 2.83 [10:42:37] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.93, 3.68, 3.27 [10:44:33] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.78, 3.16, 3.14 [10:45:32] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2083093226 secs [10:50:26] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.80, 4.06, 3.52 [10:54:19] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.38, 3.90, 3.60 [10:58:14] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.59, 3.11, 2.26 [11:00:14] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.10, 2.34, 2.08 [11:02:06] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.31, 3.82, 3.61 [11:04:02] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.20, 3.59, 3.55 [11:07:32] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08788409829 secs [11:11:48] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.30, 4.36, 3.80 [11:11:57] [02puppet] 07Reception123 commented on pull request 03#2996: allow direct pushing of certs to GitHub T7582 - 13https://github.com/miraheze/puppet/pull/2996#issuecomment-1312456192 [11:11:58] [url] allow direct pushing of certs to GitHub T7582 by Reception123 · Pull Request #2996 · miraheze/puppet · GitHub | github.com [11:14:32] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.18, 2.69, 1.89 [11:16:31] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 3.17, 2.72, 2.00 [11:41:55] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.01, 3.76, 2.37 [11:43:52] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.76, 3.00, 2.26 [11:51:52] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.92, 3.60, 3.07 [11:53:49] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.80, 3.19, 2.97 [11:56:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.43, 2.84 [11:58:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 3.10, 2.80 [12:03:04] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.49, 3.18, 2.29 [12:05:03] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.45, 2.54, 2.16 [12:16:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.53, 3.06 [12:16:40] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.46, 4.85, 3.89 [12:18:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 3.01, 2.92 [12:18:40] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.15, 3.78, 3.61 [12:18:52] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1460988522 secs [12:24:18] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.88, 2.81, 3.23 [12:36:27] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.0953117013 secs [12:46:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.85, 4.65, 3.52 [12:46:31] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.08, 3.94, 3.09 [12:48:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.38, 3.51, 3.22 [12:48:29] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.40, 3.16, 2.91 [12:50:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.62, 3.03, 3.09 [12:58:17] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.25, 3.09, 2.27 [12:58:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.72, 3.46, 3.23 [12:59:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.32, 3.24, 3.88 [13:00:11] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.63, 3.29, 2.45 [13:00:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.77, 3.86, 3.40 [13:02:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.32, 3.31, 3.25 [13:04:00] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.37, 2.78, 2.48 [13:05:44] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.82, 4.16, 4.05 [13:06:48] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.46, 3.08, 2.55 [13:07:41] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.26, 3.80, 3.93 [13:08:45] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 1.96, 2.63, 2.45 [13:09:37] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.62, 4.06, 3.98 [13:10:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.43, 3.02 [13:10:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.59, 3.72, 3.39 [13:12:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.97, 2.90, 2.88 [13:12:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.16, 3.09, 3.20 [13:15:25] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.17, 3.57, 3.86 [13:16:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 3.78, 4.20, 3.64 [13:17:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.58, 3.24 [13:18:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.31, 3.46, 3.44 [13:19:18] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.43, 4.17, 4.01 [13:20:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.46, 3.66, 3.50 [13:21:14] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.33, 3.51, 3.79 [13:21:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.43, 4.13, 3.51 [13:21:37] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:22:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.68, 3.67, 3.52 [13:23:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.46, 3.34 [13:24:36] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.55, 3.88, 3.26 [13:25:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.28, 4.02, 3.52 [13:26:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.71, 4.57, 3.87 [13:26:33] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.33, 3.24, 3.09 [13:27:08] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.41, 3.44, 3.65 [13:27:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.75, 3.48 [13:29:08] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.23, 3.16, 3.53 [13:31:08] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.61, 2.51, 3.24 [13:34:09] PROBLEM - gluster112 APT on gluster112 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:35:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 3.90, 3.65 [13:36:09] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.60, 3.64, 3.37 [13:37:08] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.81, 3.81, 3.53 [13:39:08] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.38, 3.16, 3.32 [13:40:02] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.19, 2.59, 3.03 [13:41:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.83, 3.78 [13:43:08] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.44, 4.39, 3.82 [13:45:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.37, 4.65, 4.09 [13:46:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.43, 3.43, 4.00 [13:47:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.93, 3.89 [13:51:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 4.25, 4.02 [13:53:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.49, 3.57, 3.80 [13:55:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.51, 4.55, 4.13 [13:55:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 15.17, 6.49, 3.38 [13:56:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 13.56, 6.01, 4.50 [13:57:08] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 1.25, 3.40, 3.84 [13:58:43] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:49] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.426 second response time [14:01:08] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.68, 3.80, 3.86 [14:01:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.28, 3.67, 3.13 [14:03:16] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.79, 2.65, 2.82 [14:05:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.13, 3.34, 4.00 [14:05:40] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.57, 3.10, 2.62 [14:06:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.81, 3.13, 3.96 [14:07:37] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 1.57, 2.52, 2.46 [14:09:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.17, 2.21, 3.38 [14:10:23] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 0.92, 1.92, 3.26 [14:11:08] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.75, 3.81, 3.95 [14:11:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.60, 3.71, 2.64 [14:13:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.54, 2.91, 2.48 [14:21:08] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.44, 3.94, 3.82 [14:25:08] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.78, 3.67, 3.77 [14:25:16] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.07, 3.68, 2.80 [14:27:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.67, 3.86, 2.99 [14:28:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.87, 3.39, 2.73 [14:28:39] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.66, 3.84, 2.69 [14:29:08] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.16, 2.90, 3.40 [14:29:16] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.93, 3.97, 3.12 [14:30:33] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.26, 3.46, 2.70 [14:31:16] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.05, 3.29, 2.98 [14:32:03] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.59, 3.05, 2.75 [14:32:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.02, 2.64, 2.49 [14:37:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.11, 3.93, 3.05 [14:39:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.82, 3.06, 2.83 [14:43:54] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.38, 3.93, 3.66 [14:49:13] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1027876437 secs [14:51:39] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.05, 3.67, 3.62 [14:53:35] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.96, 3.42, 3.53 [14:57:16] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.63, 4.33, 3.07 [14:59:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.86, 3.59, 2.95 [15:01:16] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.65, 3.18, 2.89 [15:03:16] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.08, 3.27, 3.40 [15:07:10] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 7.49, 4.82, 3.95 [15:15:13] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07975000143 secs [15:17:10] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:09] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.401 second response time [15:25:13] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1787673235 secs [15:33:46] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1356315315 secs [15:38:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.70, 5.05, 3.11 [15:42:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.22, 3.45, 2.89 [15:44:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.21, 2.69, 2.68 [15:47:03] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.76, 2.98, 2.49 [15:49:03] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.54, 2.36, 2.32 [15:49:13] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07796153426 secs [15:52:36] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.67, 3.64, 2.53 [15:54:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 4.24, 3.06 [15:54:35] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.32, 3.20, 2.51 [15:55:49] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.01, 3.54, 2.62 [15:56:43] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.00, 6.02, 3.53 [15:57:46] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08936366439 secs [15:58:30] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.67, 4.34, 3.12 [15:59:49] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.76, 3.78, 2.91 [16:00:35] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:47] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:27] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.62, 3.86, 3.24 [16:02:30] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.418 second response time [16:02:36] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.98, 3.19, 3.13 [16:03:49] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.77, 2.44, 2.59 [16:04:26] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.60, 3.28, 3.11 [16:05:01] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.811 second response time [16:08:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.75, 4.12, 2.97 [16:10:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.42, 3.08, 2.73 [16:12:23] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.24, 3.73, 3.27 [16:12:48] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:25] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:43] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.290 second response time [16:15:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.052 second response time [16:15:53] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw121 [16:17:07] [02ssl] 07Universal-Omega closed pull request 03#614: Remove redirects for wikis which don't point to Miraheze/have been permanently deleted - 13https://github.com/miraheze/ssl/pull/614 [16:17:08] [url] Page not found · GitHub · GitHub | github.com [16:17:09] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7cc377a1cbbd...f8b83d80fa8a [16:17:10] [url] Comparing 7cc377a1cbbd...f8b83d80fa8a · miraheze/ssl · GitHub | github.com [16:17:12] [02miraheze/ssl] 07AgentIsai 03f8b83d8 - Remove redirects for wikis which don't point to Miraheze/have been deleted (#614) [16:17:52] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [16:19:13] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.114716351 secs [16:20:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [16:20:57] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:32] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.350 second response time [16:25:47] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.07519003749 secs [16:31:37] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5936 bytes in 0.015 second response time [16:32:01] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:16] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:20] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:31] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.54, 5.02, 2.93 [16:35:39] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.857 second response time [16:36:02] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.265 second response time [16:37:09] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 2 backends are down. mw121 mw131 [16:37:13] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.317 second response time [16:37:27] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.53, 3.80, 2.73 [16:38:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.86, 3.48, 2.70 [16:39:22] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.473 second response time [16:39:26] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.34, 2.89, 2.52 [16:39:51] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw131 [16:39:58] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.30, 2.70, 2.51 [16:41:09] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [16:41:46] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [16:45:26] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.09, 4.02, 3.01 [16:46:30] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:38] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:42] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:06] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.335 second response time [16:48:40] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.901 second response time [16:48:48] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.619 second response time [16:48:55] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw122 [16:49:15] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw122 [16:49:26] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.51, 3.46, 3.08 [16:50:51] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [16:51:15] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [16:51:26] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.83, 2.98, 2.95 [16:54:26] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [16:56:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.54, 3.43, 2.29 [16:58:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.17, 2.59, 2.12 [17:00:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.394 second response time [17:00:33] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw122 [17:01:12] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1080530882 secs [17:02:29] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [17:05:13] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.09809812903 secs [17:05:31] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.623 second response time [17:05:36] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.78, 3.36, 2.73 [17:08:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.91, 3.33, 2.55 [17:08:27] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:04] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2173504531 secs [17:09:35] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.11, 3.18, 2.85 [17:09:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.04, 2.88, 2.48 [17:13:35] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.86, 3.77, 3.24 [17:15:35] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.79, 3.27, 3.11 [17:16:20] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.751 second response time [17:21:01] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:44] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:01] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.779 second response time [17:24:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.522 second response time [17:25:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.59, 4.20, 2.90 [17:26:02] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.31, 3.00, 2.23 [17:26:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 11.69, 5.45, 3.22 [17:27:10] PROBLEM - cp33 PowerDNS Recursor on cp33 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:27:11] PROBLEM - cp33 SSH on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.79, 3.77, 2.90 [17:28:02] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 3.09, 3.13, 2.39 [17:28:08] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:07] RECOVERY - cp33 SSH on cp33 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:29:09] RECOVERY - cp33 PowerDNS Recursor on cp33 is OK: DNS OK: 0.226 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [17:30:05] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.283 second response time [17:31:04] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.901 second response time [17:33:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.44, 3.12, 2.93 [17:33:51] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.49, 3.03, 2.48 [17:35:09] PROBLEM - cp33 NTP time on cp33 is CRITICAL: NTP CRITICAL: Offset 0.6034433842 secs [17:35:23] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:47] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.47, 2.85, 2.49 [17:36:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.92, 3.54, 3.72 [17:37:09] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -1.460313797e-05 secs [17:37:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.77, 4.10, 3.38 [17:38:57] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 2 backends are down. mw121 mw122 [17:39:21] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.284 second response time [17:40:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.27, 2.46, 3.25 [17:42:47] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [17:43:56] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5936 bytes in 0.017 second response time [17:45:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.42, 3.94, 3.90 [17:45:55] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.485 second response time [17:47:36] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.43, 4.32, 4.08 [17:49:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.16, 3.20, 3.70 [17:51:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.72, 2.38, 3.34 [17:54:20] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:17] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:43] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.330 second response time [17:55:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.33, 4.64, 3.55 [17:56:18] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.333 second response time [17:57:14] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.260 second response time [17:57:20] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw121 [17:57:26] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw122 [17:57:50] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.40, 3.90, 3.42 [17:58:32] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw122 [17:59:19] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [17:59:26] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [17:59:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.47, 3.08, 3.18 [18:00:32] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [18:07:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1ba7ea620a56 [18:07:10] [02miraheze/mw-config] 07Universal-Omega 031ba7ea6 - Only set `$wgThumbPath` if not using Swift [18:07:13] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [18:07:13] [url] Page not found · GitHub · GitHub | github.com [18:07:14] [02mw-config] 07Universal-Omega opened pull request 03#4981: Only set `$wgThumbPath` if not using Swift - 13https://github.com/miraheze/mw-config/pull/4981 [18:07:14] [url] Page not found · GitHub · GitHub | github.com [18:07:57] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1ba7ea620a56...c0bfe85e0f1b [18:07:57] [url] Comparing 1ba7ea620a56...c0bfe85e0f1b · miraheze/mw-config · GitHub | github.com [18:07:58] [02miraheze/mw-config] 07github-actions 03c0bfe85 - CI: lint code to MediaWiki standards [18:08:01] [02mw-config] 07github-actions[bot] synchronize pull request 03#4981: Only set `$wgThumbPath` if not using Swift - 13https://github.com/miraheze/mw-config/pull/4981 [18:08:01] [url] Page not found · GitHub · GitHub | github.com [18:08:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.99, 4.93, 3.66 [18:08:33] miraheze/mw-config - Universal-Omega the build passed. [18:09:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c0bfe85e0f1b...77576258eae4 [18:09:48] [url] Comparing c0bfe85e0f1b...77576258eae4 · miraheze/mw-config · GitHub | github.com [18:09:48] [02miraheze/mw-config] 07Universal-Omega 037757625 - Update FileBackend.php [18:09:49] [02mw-config] 07Universal-Omega synchronize pull request 03#4981: Only set `$wgThumbPath` if not using Swift - 13https://github.com/miraheze/mw-config/pull/4981 [18:09:49] [url] Page not found · GitHub · GitHub | github.com [18:10:25] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.78, 3.62, 3.33 [18:10:47] miraheze/mw-config - Universal-Omega the build passed. [18:12:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.10, 2.79, 3.06 [18:12:25] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 0.74, 1.92, 3.67 [18:14:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.22, 9.33, 8.14 [18:15:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.78, 8.34, 7.92 [18:16:25] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.26, 1.76, 3.21 [18:18:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.87, 2.40, 3.84 [18:22:57] [02mw-config] 07Universal-Omega closed pull request 03#4981: Only set `$wgThumbPath` if not using Swift - 13https://github.com/miraheze/mw-config/pull/4981 [18:22:58] [url] Page not found · GitHub · GitHub | github.com [18:23:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/108a559468d3...5f6ab6484d39 [18:23:00] [url] Comparing 108a559468d3...5f6ab6484d39 · miraheze/mw-config · GitHub | github.com [18:23:02] [02miraheze/mw-config] 07Universal-Omega 035f6ab64 - Only set `$wgThumbPath` if not using Swift (#4981) [18:23:04] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [18:23:06] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [18:23:07] ... [18:23:21] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [18:23:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.53, 3.44, 2.95 [18:23:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:35] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [18:23:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:56] miraheze/mw-config - Universal-Omega the build passed. [18:24:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.48, 1.87, 3.15 [18:25:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.80, 4.26, 3.36 [18:27:35] !log [@test131] starting deploy of {'config': True} to all [18:27:36] !log [@test131] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [18:27:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:54] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [18:31:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.29, 3.84, 3.52 [18:35:12] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.34, 2.97, 3.24 [18:42:48] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.20, 3.54, 2.74 [18:44:47] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.94, 2.89, 2.60 [18:46:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.10, 5.34, 3.70 [18:46:25] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.31, 4.27, 3.10 [18:49:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.60, 3.44, 3.31 [18:50:25] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.64, 3.53, 3.09 [18:51:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 3.77, 3.43 [18:52:25] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.38, 4.31, 3.42 [18:53:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.56, 3.40 [18:54:25] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.17, 3.75, 3.32 [18:54:53] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.19, 3.24, 2.84 [18:55:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 3.27, 3.32 [18:56:25] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.41, 2.97, 3.09 [18:57:52] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:01:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.76, 3.46 [19:02:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.43, 3.84, 3.52 [19:03:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 3.26, 3.31 [19:04:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.09, 2.88, 3.20 [19:06:25] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.00, 4.13, 3.53 [19:08:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.06, 3.44, 2.55 [19:08:25] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.10, 3.63, 3.41 [19:11:35] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.41, 2.87, 2.66 [19:11:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.16, 3.18 [19:13:35] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.89, 2.46, 2.53 [19:13:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.28, 3.00, 3.10 [19:13:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 3.26, 2.82 [19:14:25] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.96, 4.28, 3.72 [19:17:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.96, 3.74, 3.50 [19:17:57] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.09, 3.53, 2.87 [19:19:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.74, 3.21, 3.34 [19:19:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.75, 3.21, 2.83 [19:20:25] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.99, 3.75, 3.80 [19:24:25] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.97, 2.77, 3.38 [19:26:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.23, 3.52, 3.05 [19:28:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.50, 2.89, 2.88 [19:34:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.76, 3.54, 3.07 [19:36:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.36, 3.53, 3.13 [19:38:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.48, 4.38, 3.45 [19:41:35] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.89, 3.43, 2.94 [19:43:35] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.79, 3.17, 2.90 [19:44:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.03, 3.99, 3.95 [19:48:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.88, 2.61, 3.37 [19:49:16] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.43, 4.32, 3.30 [19:51:11] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.66, 3.84, 3.23 [19:54:30] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 7.32, 4.41, 3.50 [19:55:01] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.30, 4.68, 3.63 [19:56:29] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.39, 3.80, 3.37 [19:57:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.08, 4.55, 3.48 [19:58:28] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.82, 3.23, 3.22 [20:09:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.13, 3.29, 3.78 [20:10:35] !log [@mwtask141] starting deploy of {'config': True} to all [20:10:41] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [20:10:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:10:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:25] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.31, 2.92, 3.84 [20:14:25] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.00, 3.96, 4.08 [20:15:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 4.16, 3.93 [20:16:25] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.73, 3.67, 3.96 [20:17:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.08, 3.39, 3.68 [20:21:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.02, 2.27, 3.15 [20:22:25] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 0.88, 2.02, 3.13 [20:31:01] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.54, 3.16, 2.93 [20:32:59] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 3.15, 3.06, 2.92 [20:36:55] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.54, 4.23, 3.40 [20:38:55] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.35, 3.65, 3.29 [20:39:05] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.115144074 secs [20:39:38] !log install python3-prometheus-client on swiftobject111 [20:39:47] !log [@mwtask141] starting deploy of {'config': True} to all [20:39:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:39:55] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [20:39:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:50] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.04, 3.13, 3.21 [20:47:01] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.92, 3.52, 3.09 [20:48:46] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.78, 3.27, 3.23 [20:48:55] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.28, 3.07, 2.97 [20:50:45] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.50, 2.71, 3.04 [20:56:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.42, 3.84, 3.33 [20:57:05] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07705050707 secs [20:57:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.47, 3.44, 3.94 [20:58:34] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.31, 4.32, 3.59 [20:59:56] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.36, 3.73, 3.99 [21:00:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.70, 3.22 [21:03:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.07, 3.18, 3.74 [21:06:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.62, 3.60, 3.40 [21:07:56] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.60, 3.53, 3.72 [21:08:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.79, 3.48 [21:09:49] !log [@mwtask141] starting deploy of {'config': True} to all [21:09:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.48, 3.43, 3.66 [21:10:52] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 63s [21:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:05] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4981-Universal-Omega-patch-2 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/9f5c83b22cc3 [21:12:55] [02miraheze/mw-config] 07Universal-Omega 039f5c83b - Revert "Only set `$wgThumbPath` if not using Swift (#4981)" [21:12:58] [02mw-config] 07Universal-Omega created branch 03revert-4981-Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [21:12:58] [url] Page not found · GitHub · GitHub | github.com [21:12:58] [02mw-config] 07Universal-Omega opened pull request 03#4982: Revert "Only set `$wgThumbPath` if not using Swift" - 13https://github.com/miraheze/mw-config/pull/4982 [21:12:59] [url] Page not found · GitHub · GitHub | github.com [21:13:09] [02mw-config] 07Universal-Omega closed pull request 03#4982: Revert "Only set `$wgThumbPath` if not using Swift" - 13https://github.com/miraheze/mw-config/pull/4982 [21:13:10] ... [21:13:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±0] 13https://github.com/miraheze/mw-config/compare/ae60db137231...3e12efd3e136 [21:13:11] [url] Comparing ae60db137231...3e12efd3e136 · miraheze/mw-config · GitHub | github.com [21:13:11] [02miraheze/mw-config] 07Universal-Omega 033e12efd - Revert "Only set `$wgThumbPath` if not using Swift" (#4982) [21:13:14] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4981-Universal-Omega-patch-2 [21:13:15] [02mw-config] 07Universal-Omega deleted branch 03revert-4981-Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [21:13:16] [url] Page not found · GitHub · GitHub | github.com [21:13:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b4d35d828993...fd18116d02f1 [21:13:51] [url] Comparing b4d35d828993...fd18116d02f1 · miraheze/puppet · GitHub | github.com [21:13:53] [02miraheze/puppet] 07Universal-Omega 03fd18116 - swift-rewrite: fix container used for thumb [21:13:56] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 2.95, 4.11, 3.97 [21:13:57] miraheze/mw-config - Universal-Omega the build passed. [21:14:19] miraheze/mw-config - Universal-Omega the build passed. [21:15:27] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [21:15:46] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.21, 3.59, 3.80 [21:16:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:16:09] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 41s [21:16:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:16:18] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.791 second response time [21:16:55] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:43] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.974 second response time [21:17:56] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.35, 4.00, 3.90 [21:18:52] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.325 second response time [21:19:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.01, 3.14, 2.70 [21:21:06] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.37, 3.22, 2.79 [21:21:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.99, 3.91, 3.90 [21:23:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2966f4676ffb [21:23:47] [02miraheze/puppet] 07paladox 032966f46 - swift: remove container as a param from handle404 [21:23:48] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [21:23:49] [url] Page not found · GitHub · GitHub | github.com [21:23:51] [02puppet] 07paladox opened pull request 03#2997: swift: remove container as a param from handle404 - 13https://github.com/miraheze/puppet/pull/2997 [21:23:52] [url] Page not found · GitHub · GitHub | github.com [21:25:44] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 15.73, 6.63, 3.70 [21:25:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.99, 2.96, 3.96 [21:26:34] !log [@test131] starting deploy of {'config': True} to all [21:26:35] !log [@test131] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [21:26:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:26:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:56] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 1.85, 2.46, 3.24 [21:29:53] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [21:33:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.62, 3.50, 3.52 [21:33:56] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.78, 4.47, 3.79 [21:35:35] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.35, 3.02, 3.33 [21:35:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 2.76, 3.34 [21:41:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.00, 3.80, 3.58 [21:43:04] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.46, 3.42, 2.96 [21:43:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.90, 3.44, 3.36 [21:43:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.48, 3.48 [21:43:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.33, 3.87, 3.82 [21:45:56] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.70, 4.39, 4.00 [21:46:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.78, 2.86, 2.46 [21:47:04] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.72, 3.14, 2.96 [21:47:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.78, 3.30, 3.36 [21:47:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 4.13, 3.75 [21:48:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.65, 2.57, 2.40 [21:49:01] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:00] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.582 second response time [21:51:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.93, 3.84 [21:53:04] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.14, 3.47, 3.11 [21:53:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 4.26, 3.97 [21:55:04] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.99, 3.31, 3.10 [21:55:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.91, 3.88 [21:57:09] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.07, 2.48, 1.69 [21:57:52] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [21:58:53] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.80, 3.27, 2.89 [21:59:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.31, 2.05, 1.64 [21:59:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.64, 3.74 [22:00:53] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.50, 2.92, 2.80 [22:01:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.39, 3.63 [22:03:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.55, 2.78, 3.39 [22:05:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.05, 3.38, 3.94 [22:06:25] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.48, 2.90, 3.84 [22:08:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.38, 3.55, 2.92 [22:10:07] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.75, 5.89, 3.92 [22:10:25] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.93, 2.12, 3.28 [22:13:56] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 6.18, 3.74, 3.71 [22:14:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.12, 3.15, 3.23 [22:17:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.60, 3.14, 2.95 [22:17:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.46, 3.43, 3.63 [22:19:47] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.07, 2.72, 2.82 [22:21:56] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.49, 2.94, 3.38 [22:28:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.34, 3.50, 2.80 [22:30:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.60, 2.67, 2.57 [22:37:56] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.53, 3.72, 3.39 [22:42:25] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.06, 3.56, 3.18 [22:44:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.48, 4.17, 3.30 [22:44:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 3.59, 2.84 [22:46:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.91, 3.67, 3.22 [22:46:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.99, 2.94, 2.69 [22:48:06] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.65, 3.31, 3.15 [22:50:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.62, 3.06 [22:52:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.67, 4.24, 3.33 [22:58:32] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:09] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5936 bytes in 0.011 second response time [22:59:14] PROBLEM - gluster102 PowerDNS Recursor on gluster102 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:59:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:58] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:01] PROBLEM - gluster101 SSH on gluster101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:06] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:13] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:13] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:14] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:25] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:43] PROBLEM - cloud10 Current Load on cloud10 is CRITICAL: CRITICAL - load average: 82.68, 51.08, 29.11 [23:03:54] PROBLEM - gluster101 PowerDNS Recursor on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:04:02] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:03] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [23:04:11] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:21] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:31] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [23:05:15] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [23:05:26] PROBLEM - gluster101 Puppet on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:05:26] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [23:05:36] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [23:05:47] PROBLEM - gluster102 ferm_active on gluster102 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:05:50] PROBLEM - gluster102 Puppet on gluster102 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:05:50] PROBLEM - gluster101 ferm_active on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:05:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 0.14, 2.14, 3.77 [23:06:05] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 52% [23:06:25] PROBLEM - gluster101 NTP time on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:06:26] PROBLEM - gluster102 Disk Space on gluster102 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:06:36] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 40% [23:07:56] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 0.13, 1.46, 3.32 [23:08:01] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [23:08:22] PROBLEM - gluster101 Disk Space on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:08:57] PROBLEM - gluster101 glusterd Volume on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:08:58] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.96, 8.58, 5.95 [23:09:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.51, 9.08, 6.74 [23:09:12] PROBLEM - gluster101 glusterd on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:09:24] PROBLEM - gluster101 conntrack_table_size on gluster101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:09:43] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 17.96, 10.36, 6.33 [23:10:29] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 34% [23:10:41] PROBLEM - gluster102 NTP time on gluster102 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:10:56] PROBLEM - gluster102 glusterd on gluster102 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:11:26] PROBLEM - gluster102 conntrack_table_size on gluster102 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:11:57] PROBLEM - gluster102 glusterd Volume on gluster102 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:12:03] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 52% [23:12:58] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.49, 10.25, 7.31 [23:13:10] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.397 second response time [23:13:37] !log stopped and started gluster101, some how frozed causing high cpu to sky rocket including i/o [23:13:44] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.014 second response time [23:14:03] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [23:14:24] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 58% [23:14:28] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.015 second response time [23:14:44] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.411 second response time [23:14:58] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 1.57, 7.00, 6.47 [23:15:01] !log stopped and started gluster102, some how frozed causing high cpu to sky rocket including i/o [23:16:03] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 57% [23:16:03] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.019 second response time [23:16:05] PROBLEM - Host gluster101 is DOWN: PING CRITICAL - Packet loss = 100% [23:16:20] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 29% [23:16:24] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.862 second response time [23:16:29] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [23:16:38] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.199 second response time [23:17:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.23, 11.88, 9.50 [23:17:38] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.014 second response time [23:17:38] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.356 second response time [23:17:38] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.019 second response time [23:17:40] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 10% [23:17:50] PROBLEM - gluster102 SSH on gluster102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:51] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.015 second response time [23:18:00] PROBLEM - ping6 on gluster102 is CRITICAL: PING CRITICAL - Packet loss = 100% [23:18:03] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 9% [23:18:04] RECOVERY - Host gluster101 is UP: PING OK - Packet loss = 0%, RTA = 0.96 ms [23:18:43] RECOVERY - gluster102 glusterd on gluster102 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterd' [23:19:06] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:19:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 1.00, 7.96, 8.35 [23:19:12] RECOVERY - gluster101 Disk Space on gluster101 is OK: DISK OK - free space: / 139825 MB (14% inode=81%); [23:19:12] RECOVERY - gluster101 glusterd on gluster101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterd' [23:19:12] RECOVERY - gluster102 conntrack_table_size on gluster102 is OK: OK: nf_conntrack is 0 % full [23:19:18] RECOVERY - gluster101 conntrack_table_size on gluster101 is OK: OK: nf_conntrack is 0 % full [23:19:19] RECOVERY - gluster101 SSH on gluster101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:19:19] RECOVERY - gluster101 glusterd Volume on gluster101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfsd' [23:19:28] RECOVERY - gluster101 NTP time on gluster101 is OK: NTP OK: Offset -0.003996998072 secs [23:19:29] RECOVERY - gluster101 ferm_active on gluster101 is OK: OK ferm input default policy is set [23:19:33] RECOVERY - gluster102 Puppet on gluster102 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [23:19:42] RECOVERY - gluster102 ferm_active on gluster102 is OK: OK ferm input default policy is set [23:19:43] PROBLEM - cloud10 Current Load on cloud10 is WARNING: WARNING - load average: 13.20, 38.01, 46.01 [23:19:43] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 1.43, 9.12, 9.00 [23:19:46] RECOVERY - gluster102 SSH on gluster102 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:19:46] RECOVERY - gluster102 glusterd Volume on gluster102 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfsd' [23:19:49] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.69, 1.17, 0.44 [23:19:50] RECOVERY - gluster101 PowerDNS Recursor on gluster101 is OK: DNS OK: 0.148 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [23:20:00] RECOVERY - ping6 on gluster102 is OK: PING OK - Packet loss = 0%, RTA = 1.36 ms [23:20:15] PROBLEM - gluster102 Disk Space on gluster102 is WARNING: DISK WARNING - free space: / 55314 MB (10% inode=72%); [23:20:20] RECOVERY - gluster102 PowerDNS Recursor on gluster102 is OK: DNS OK: 0.206 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [23:20:29] RECOVERY - gluster102 NTP time on gluster102 is OK: NTP OK: Offset -6.41644001e-05 secs [23:20:32] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [23:20:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.18, 3.31, 1.53 [23:21:15] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [23:21:26] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [23:21:31] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [23:23:43] RECOVERY - cloud10 Current Load on cloud10 is OK: OK - load average: 5.88, 20.49, 36.97 [23:29:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.79, 4.29, 2.73 [23:31:01] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.42, 3.60, 2.67 [23:32:59] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 3.08, 2.59 [23:52:06] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.97, 3.73, 2.89 [23:52:39] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.25, 3.90, 3.29 [23:54:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.44, 3.94, 3.06 [23:54:36] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.39, 4.01, 3.39 [23:55:58] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.10, 3.37, 2.96 [23:56:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.78, 3.51, 3.28 [23:58:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.13, 3.07, 3.14