[00:01:39] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 37.74, 16.81, 7.58 [00:01:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.18, 11.64, 11.89 [00:02:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.15, 4.12, 2.93 [00:04:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.03, 3.24, 2.74 [00:05:22] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.86, 3.46, 3.16 [00:05:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 11.46, 11.81 [00:06:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.76, 11.27, 11.96 [00:07:18] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.18, 3.26, 3.12 [00:07:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.78, 11.83, 11.89 [00:08:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.77, 11.66, 12.04 [00:09:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.62, 11.22, 11.68 [00:10:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.18, 11.62, 11.98 [00:12:13] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.47, 3.71, 3.80 [00:12:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.67, 12.20, 12.14 [00:14:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.76, 4.56, 4.11 [00:14:30] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.25, 3.48, 3.29 [00:15:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.01, 10.76, 11.23 [00:16:29] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.12, 3.21, 3.20 [00:17:38] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.42, 1.56, 3.63 [00:17:55] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.52, 3.29, 3.14 [00:18:21] RECOVERY - widedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'widedata.org' will expire on Fri 24 Feb 2023 05:04:05 GMT +0000. [00:19:38] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.62, 1.25, 3.26 [00:19:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.68, 11.56, 11.49 [00:19:53] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.17, 3.08, 3.07 [00:22:14] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.14, 3.84, 3.98 [00:32:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.01, 10.82, 11.84 [00:34:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.05, 3.71, 3.75 [00:34:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.66, 11.88, 12.12 [00:35:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.23, 12.58, 11.59 [00:36:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.64, 3.31, 3.60 [00:36:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.47, 11.35, 11.90 [00:41:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.74, 10.96, 11.51 [00:42:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.11, 11.14, 11.39 [00:46:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.24, 3.65, 3.55 [00:46:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.80, 11.96, 11.73 [00:47:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.11, 11.60, 11.47 [00:48:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.07, 3.39, 3.46 [00:49:48] PROBLEM - widedata.org - LetsEncrypt on sslhost is CRITICAL: connect to address widedata.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:55:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.60, 4.25, 2.53 [00:57:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.83, 3.89, 2.61 [00:57:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.77, 11.93, 11.94 [00:58:11] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.77, 3.15, 3.34 [00:59:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.11, 3.29, 2.55 [00:59:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.94, 12.16, 12.01 [01:01:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.27, 11.67, 11.87 [01:03:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.83, 11.86, 11.90 [01:06:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.83, 8.61, 9.97 [01:12:07] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.26, 3.45, 3.31 [01:12:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.19, 11.19, 10.61 [01:12:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.54, 3.38, 3.18 [01:13:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.30, 10.56, 11.63 [01:14:06] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.30, 3.43, 3.32 [01:14:22] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.13, 3.13, 3.10 [01:16:05] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.08, 3.31, 3.28 [01:16:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.91, 11.78, 11.07 [01:19:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.50, 11.08, 11.43 [01:20:03] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.69, 3.95, 3.57 [01:21:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.42, 11.15, 11.43 [01:23:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.20, 12.21, 11.78 [01:24:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.13, 11.87, 11.22 [01:25:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.74, 11.08, 11.41 [01:26:01] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.31, 3.99, 3.69 [01:26:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.74, 11.50, 11.17 [01:30:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.44, 11.72, 11.31 [01:32:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.06, 11.03, 11.11 [01:33:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.76, 11.71, 11.39 [01:33:59] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.38, 3.73, 3.73 [01:40:12] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 16.39, 9.84, 6.92 [01:41:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.23, 11.74, 11.68 [01:41:56] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.80, 2.96, 3.36 [01:42:12] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.86, 8.14, 6.66 [01:42:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.70, 11.92, 11.21 [01:44:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.70, 11.56, 11.16 [01:45:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.35, 3.33, 3.42 [01:47:52] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.20, 3.27, 3.38 [01:52:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.15, 11.61, 11.35 [01:53:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.03, 11.29, 11.12 [01:54:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.46, 11.03, 11.16 [01:55:05] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.42, 3.31, 2.08 [01:57:04] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.81, 2.95, 2.09 [01:57:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.30, 10.58, 10.95 [01:58:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.73, 12.17, 11.54 [02:00:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.97, 11.20, 11.25 [02:05:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.65, 12.13, 11.38 [02:12:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.67, 12.21, 11.39 [02:13:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.13, 11.51, 11.52 [02:17:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.96, 11.92, 11.66 [02:19:37] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.06, 3.79, 3.47 [02:20:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.88, 11.33, 11.61 [02:21:36] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.51, 3.77, 3.50 [02:21:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.42, 11.87, 11.75 [02:26:12] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.14, 8.20, 6.32 [02:28:12] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.31, 6.96, 6.09 [02:29:33] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.07, 3.26, 3.36 [02:29:38] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.03, 3.65, 1.80 [02:29:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.59, 10.51, 10.88 [02:31:38] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.94, 3.86, 2.13 [02:33:38] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.85, 4.70, 2.68 [02:35:38] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.80, 3.21, 2.37 [02:38:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.72, 8.89, 10.11 [02:39:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.38, 10.95, 11.29 [02:41:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.18, 11.97, 11.58 [02:42:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.12, 9.85, 10.19 [02:43:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.49, 11.85, 11.60 [02:44:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.53, 11.30, 10.69 [02:45:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.38, 11.87, 11.62 [02:46:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.50, 11.19, 10.75 [02:47:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 10.95, 11.32 [02:47:56] RECOVERY - widedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'widedata.org' will expire on Fri 24 Feb 2023 05:04:05 GMT +0000. [02:52:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.66, 9.04, 9.93 [02:59:38] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.60, 2.84, 1.83 [02:59:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.44, 11.85, 11.26 [03:00:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.86, 10.76, 10.23 [03:01:38] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.43, 2.92, 2.00 [03:01:47] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.61, 2.56, 1.91 [03:02:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.89, 9.81, 9.96 [03:03:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 10.65, 10.98 [03:05:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.97, 2.84, 2.24 [03:11:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.27, 8.72, 10.16 [03:13:15] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.55, 3.56, 3.36 [03:15:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.20, 10.05, 10.43 [03:17:13] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.73, 3.25, 3.29 [03:17:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.89, 8.84, 9.94 [03:19:23] PROBLEM - widedata.org - LetsEncrypt on sslhost is CRITICAL: connect to address widedata.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:20:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.83, 3.53, 3.17 [03:22:04] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.32, 3.14, 3.07 [03:33:06] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 4.00, 3.81, 3.49 [03:35:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.76, 10.94, 9.81 [03:36:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1174867153 secs [03:37:05] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.64, 3.33, 3.37 [03:37:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.64, 11.16, 10.04 [03:43:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.33, 10.60, 9.65 [03:43:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.31, 8.95, 9.48 [03:45:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.31, 10.07, 9.56 [03:47:59] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.35, 3.42, 3.39 [03:49:58] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.66, 3.18, 3.30 [03:55:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.00, 3.07, 1.98 [03:57:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.14, 2.61, 1.95 [03:59:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.37, 10.07, 9.41 [04:00:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.40, 3.69, 3.40 [04:01:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.27, 9.28, 9.20 [04:02:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.78, 3.41, 3.33 [04:04:51] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.42, 3.14, 3.25 [04:06:09] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08890199661 secs [04:08:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.15, 4.29, 2.78 [04:12:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.29, 3.54, 2.86 [04:14:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.96, 3.12, 2.79 [04:17:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.29, 3.44, 3.36 [04:19:43] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.87, 3.27, 3.31 [04:24:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.19, 10.23, 9.04 [04:24:39] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.20, 3.59, 3.08 [04:25:41] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.29, 3.68, 3.50 [04:26:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.57, 9.62, 8.96 [04:26:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.52, 3.14, 2.98 [04:27:40] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.67, 3.32, 3.39 [04:36:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.33, 3.47, 2.58 [04:44:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.61, 3.63, 3.48 [04:44:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.51, 10.40, 9.12 [04:45:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.96, 3.40, 3.28 [04:46:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.04, 2.75, 3.17 [04:46:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.08, 10.22, 9.21 [04:47:30] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.18, 3.38, 3.29 [04:48:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.74, 9.69, 9.14 [05:08:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.02, 3.73, 2.86 [05:10:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.31, 2.74, 2.60 [05:25:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.51, 3.77, 2.32 [05:27:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.93, 3.94, 2.56 [05:31:30] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.49, 3.07, 2.57 [05:45:59] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.00, 11.01, 6.11 [05:46:10] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1041 bytes in 0.159 second response time [05:46:13] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [05:47:01] RECOVERY - widedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'widedata.org' will expire on Fri 24 Feb 2023 05:04:05 GMT +0000. [05:47:58] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 1.08, 7.36, 5.36 [05:49:58] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.26, 5.00, 4.74 [05:50:50] !log [void@db121] restart mysql (OOM) [05:50:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:50] !log [void@db121] SET GLOBAL read_only=0 [05:51:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:52:10] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.491 second response time [05:52:13] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 127 Threads: 11 Questions: 43011 Slow queries: 3399 Opens: 2609 Open tables: 2603 Queries per second avg: 338.669 [05:59:38] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.66, 3.89, 1.89 [06:00:56] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.99, 3.53, 3.13 [06:01:38] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.04, 3.75, 2.11 [06:03:38] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.77, 2.68, 1.92 [06:04:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.43, 3.16, 3.09 [06:09:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.73, 3.46, 2.87 [06:10:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.71, 3.62, 3.31 [06:11:20] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.23, 3.23, 2.85 [06:15:14] PROBLEM - mw131 Puppet on mw131 is WARNING: WARNING: Puppet last ran 1 hour ago [06:16:49] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.11, 3.35, 3.29 [06:18:31] PROBLEM - widedata.org - LetsEncrypt on sslhost is CRITICAL: connect to address widedata.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:27:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.42, 4.73, 2.72 [06:27:39] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.02, 2.88, 1.79 [06:29:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.39, 3.48, 2.50 [06:31:30] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.84, 2.56, 2.28 [06:35:38] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.95, 3.26, 2.76 [06:36:40] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.20, 3.44, 3.31 [06:38:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.78, 4.21, 2.61 [06:38:40] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.49, 3.80, 3.45 [06:39:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.74, 3.34, 2.89 [06:40:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.29, 3.34, 2.47 [06:40:39] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.75, 3.39, 3.34 [06:41:40] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.93, 3.08, 2.84 [06:43:14] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:44:37] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.11, 3.75, 3.49 [06:45:37] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.21, 3.58, 3.15 [06:46:36] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.76, 3.76, 3.52 [06:47:36] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 6.76, 4.36, 3.46 [06:48:35] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.58, 4.10, 3.67 [06:51:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.38, 3.76, 3.46 [06:54:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.46, 3.98, 3.77 [06:55:33] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.45, 3.20, 3.30 [06:56:01] !log [universalomega@test131] starting deploy of {'pull': 'config', 'branch': 'Universal-Omega-patch-2', 'config': True} to all [06:56:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:56:12] !log [universalomega@test131] finished deploy of {'pull': 'config', 'branch': 'Universal-Omega-patch-2', 'config': True} to all - SUCCESS in 11s [06:56:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:56:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.78, 4.25, 3.89 [06:56:59] !log [@test131] starting deploy of {'config': True} to all [06:57:00] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:57:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1027795076 secs [06:58:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.90, 3.72, 3.74 [07:01:17] [02mw-config] 07Universal-Omega closed pull request 03#5015: Add support for database maintenance error pages - 13https://github.com/miraheze/mw-config/pull/5015 [07:01:18] [url] Page not found · GitHub · GitHub | github.com [07:01:18] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [07:01:19] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [07:01:20] [url] Page not found · GitHub · GitHub | github.com [07:01:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/416c207e02a4...6575a3485dc2 [07:01:22] [url] Comparing 416c207e02a4...6575a3485dc2 · miraheze/mw-config · GitHub | github.com [07:01:23] [02miraheze/mw-config] 07Universal-Omega 036575a34 - Add support for database maintenance error pages (#5015) [07:01:40] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [07:01:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:01:48] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 7s [07:01:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:22] miraheze/mw-config - Universal-Omega the build passed. [07:02:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.07, 3.78, 3.75 [07:04:29] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.02, 3.46, 3.64 [07:05:22] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/959bcc7dff78...e231ce21b88a [07:05:23] [url] Comparing 959bcc7dff78...e231ce21b88a · miraheze/ErrorPages · GitHub | github.com [07:05:23] [02miraheze/ErrorPages] 07Universal-Omega 03e231ce2 - databaseMaintenance: use different icon [07:05:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.65, 9.59, 8.20 [07:06:28] miraheze/ErrorPages - Universal-Omega the build passed. [07:06:30] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/e231ce21b88a...a4484ceb41e3 [07:06:30] [url] Comparing e231ce21b88a...a4484ceb41e3 · miraheze/ErrorPages · GitHub | github.com [07:06:32] [02miraheze/ErrorPages] 07Universal-Omega 03a4484ce - databaseMaintenance: use 503 response code [07:07:19] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.09, 3.34, 3.03 [07:07:21] miraheze/ErrorPages - Universal-Omega the build passed. [07:07:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.33, 9.47, 8.35 [07:08:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.20, 3.80, 2.40 [07:08:10] !log [universalomega@test131] starting deploy of {'pull': 'config', 'config': True} to all [07:08:11] !log [universalomega@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [07:08:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:19] !log [universalomega@test131] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [07:08:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:20] !log [universalomega@test131] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 0s [07:08:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:08] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=zhpolcompballwiki --no-updates --username-prefix=mh:zhpolcompball /home/universalomega/ImportDump/zhpolcompballwiki-20221218005549.xml (START) [07:09:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:15] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.28, 3.12, 2.97 [07:10:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.26, 3.54, 3.60 [07:12:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.02, 3.34, 3.52 [07:14:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.28, 3.41, 2.94 [07:16:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.63, 2.50, 2.66 [07:16:24] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.50, 2.94, 3.32 [07:19:44] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=zhpolcompballwiki --no-updates --username-prefix=mh:zhpolcompball /home/universalomega/ImportDump/zhpolcompballwiki-20221218005549.xml (END - exit=0) [07:19:45] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=zhpolcompballwiki (START) [07:19:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:25:02] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=zhpolcompballwiki (END - exit=0) [07:25:03] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=zhpolcompballwiki --active --update (END - exit=0) [07:25:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:25:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:30:08] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08926948905 secs [07:38:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.07, 4.40, 2.92 [07:42:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.51, 3.63, 3.01 [07:44:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.96, 2.78, 2.77 [07:46:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.71, 3.61, 3.24 [07:48:11] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.67, 3.19, 3.14 [07:59:38] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.54, 3.44, 1.77 [08:03:38] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.42, 3.04, 2.06 [08:17:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.61, 3.65, 3.14 [08:23:17] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.57, 3.49, 3.27 [08:23:59] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.69, 3.59, 3.14 [08:25:15] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.85, 3.40, 3.27 [08:25:58] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.70, 3.28, 3.08 [08:45:49] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.02, 3.54, 3.15 [08:47:49] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.23, 3.47, 3.17 [08:49:48] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.29, 3.06, 3.06 [09:15:37] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.86, 4.88, 3.77 [09:16:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.87, 4.21, 3.49 [09:19:36] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.81, 3.80, 3.58 [09:20:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.50, 3.71, 3.45 [09:24:22] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.96, 3.31, 3.34 [09:25:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.06, 3.56, 3.52 [09:27:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.87, 3.26, 3.41 [09:28:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.31, 3.43, 3.39 [09:29:32] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.37, 3.04, 3.32 [09:32:22] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.07, 3.31, 3.36 [09:36:14] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.86, 3.07, 2.79 [09:38:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.89, 2.95, 2.78 [09:41:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.26, 3.53, 3.32 [09:43:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.72, 3.69, 3.41 [09:47:13] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.83, 3.50, 3.36 [09:47:24] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.68, 3.36, 3.37 [09:49:11] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.65, 3.22, 3.28 [09:55:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.97, 3.64, 3.43 [09:57:02] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.72, 4.12, 3.64 [09:59:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.90, 3.72, 3.55 [09:59:18] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.11, 3.83, 3.56 [10:01:17] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.77, 3.50, 3.47 [10:06:40] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.95, 3.16, 2.74 [10:10:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.33, 10.10, 8.48 [10:10:32] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.32, 3.26, 2.87 [10:11:13] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.83, 3.26, 3.39 [10:12:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.34, 8.83, 8.21 [10:14:45] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.03, 3.51, 3.48 [10:15:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.22, 3.82, 3.58 [10:17:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.51, 3.62, 3.53 [10:18:41] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.81, 3.62, 3.53 [10:22:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.71, 3.25, 3.40 [10:23:08] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.80, 3.17, 3.36 [10:33:02] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.20, 3.54, 3.46 [10:35:01] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.12, 3.28, 3.36 [10:36:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.17, 10.24, 8.72 [10:38:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.31, 9.21, 8.54 [10:47:14] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.41, 3.92, 3.40 [10:48:55] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.42, 3.46, 3.34 [10:49:12] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.24, 3.68, 3.37 [10:56:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.68, 4.13, 3.63 [10:59:03] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.24, 3.03, 3.26 [11:00:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.94, 4.00, 3.68 [11:12:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.78, 3.70, 3.57 [11:14:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.99, 3.80, 3.62 [11:16:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.11, 3.86, 3.66 [11:17:42] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.65, 3.82, 3.42 [11:19:40] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.42, 4.04, 3.54 [11:21:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.51, 3.91, 3.56 [11:22:41] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.15, 3.93, 3.81 [11:27:33] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.90, 3.14, 3.31 [11:28:39] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.08, 2.81, 3.35 [11:49:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.14, 3.43, 2.93 [11:51:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.95, 3.31, 2.95 [11:57:11] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.38, 2.70, 1.54 [11:59:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.74, 3.06, 1.82 [12:00:24] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.13, 3.76, 3.49 [12:01:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.69, 3.23, 2.05 [12:02:24] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.68, 3.67, 3.49 [12:04:23] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.60, 4.01, 3.63 [12:05:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.54, 11.66, 8.96 [12:06:22] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.30, 3.75, 3.59 [12:06:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.86, 3.55, 3.35 [12:08:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.30, 3.61, 3.38 [12:09:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.22, 11.60, 9.58 [12:10:42] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.78, 3.26, 3.27 [12:11:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.70, 10.20, 9.31 [12:12:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.48, 10.52, 9.30 [12:16:18] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.24, 3.04, 3.31 [12:16:34] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.56, 3.99, 3.53 [12:16:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.19, 9.94, 9.35 [12:18:33] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.63, 3.50, 3.41 [12:20:31] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.47, 3.15, 3.29 [12:24:25] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.42, 3.43, 3.38 [12:25:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.94, 3.59, 3.23 [12:25:13] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.95, 3.52, 3.36 [12:25:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.01, 10.69, 9.90 [12:26:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.28, 9.84, 9.45 [12:26:23] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.16, 3.39, 3.37 [12:27:04] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.40, 3.14, 3.10 [12:27:13] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.05, 3.36, 3.32 [12:28:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.41, 9.25, 9.28 [12:29:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.48, 9.78, 9.72 [12:30:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.79, 3.57, 3.46 [12:32:34] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.07, 9.13, 4.56 [12:34:22] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.65, 3.27, 3.37 [12:38:21] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.76, 3.34, 3.35 [12:40:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.13, 9.47, 9.08 [12:41:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.08, 10.54, 9.94 [12:42:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.54, 3.15, 1.93 [12:42:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.71, 8.55, 8.77 [12:43:04] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.08, 3.78, 3.45 [12:43:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 9.48, 9.62 [12:44:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.53, 2.45, 1.81 [12:47:03] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.84, 3.89, 3.58 [12:48:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.25, 3.75, 3.54 [12:53:00] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.17, 3.86, 3.59 [12:55:00] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.08, 3.51, 3.50 [12:56:22] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.36, 3.33, 3.40 [13:06:51] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.56, 3.24, 2.98 [13:08:47] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.65, 3.09, 2.96 [13:10:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.60, 9.87, 8.52 [13:10:53] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.24, 3.03, 3.32 [13:12:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.73, 10.89, 9.09 [13:14:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 6.10, 4.16, 3.65 [13:16:33] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.53, 3.54, 3.16 [13:16:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 5.49, 3.56, 2.93 [13:17:14] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.02, 4.86, 3.85 [13:18:29] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.62, 3.33, 3.13 [13:18:40] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.86, 3.79, 3.10 [13:20:48] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.22, 3.78, 3.66 [13:21:10] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.61, 3.99, 3.76 [13:22:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.13, 3.93, 3.72 [13:24:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 10.27, 9.18 [13:24:37] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 1.98, 2.88, 2.93 [13:24:46] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.40, 3.79, 3.70 [13:25:08] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.65, 4.01, 3.81 [13:25:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.13, 9.95, 9.19 [13:26:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.56, 9.75, 9.67 [13:27:06] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.24, 3.62, 3.68 [13:33:01] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.63, 3.72, 3.62 [13:33:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.36, 10.11, 9.49 [13:34:59] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.29, 3.46, 3.54 [13:35:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.05, 10.61, 9.74 [13:36:41] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.81, 3.14, 3.36 [13:37:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 10.71, 9.94 [13:39:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.40, 10.37, 10.00 [13:40:39] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.54, 3.73, 3.55 [13:41:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.71, 11.33, 10.38 [13:41:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.92, 9.63, 9.67 [13:42:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 10.54, 10.21 [13:44:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.34, 3.76, 3.62 [13:44:49] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.74, 4.04, 3.69 [13:46:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.11, 3.67, 3.60 [13:48:37] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.22, 3.78, 3.64 [13:48:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.84, 9.33, 9.90 [13:50:36] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.17, 3.59, 3.59 [13:56:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.06, 3.23, 3.38 [13:58:35] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.32, 3.72, 3.61 [14:00:30] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.13, 3.33, 3.42 [14:01:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.31, 10.06, 9.50 [14:02:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.02, 3.58, 3.50 [14:02:31] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.75, 4.29, 2.82 [14:04:27] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.64, 3.39, 3.42 [14:04:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.73, 3.99, 3.78 [14:06:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.65, 9.51, 9.34 [14:06:22] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.09, 3.17, 2.74 [14:06:25] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.59, 3.84, 3.58 [14:07:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.16, 11.86, 10.46 [14:08:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.60, 8.84, 9.10 [14:09:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.15, 11.19, 10.42 [14:10:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.83, 5.32, 2.76 [14:10:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 4.00, 3.77, 3.60 [14:12:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.62, 10.89, 9.93 [14:12:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.53, 4.02, 3.69 [14:14:29] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.18, 3.89, 3.75 [14:15:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.02, 10.12, 10.20 [14:16:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.37, 3.15, 2.69 [14:16:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.05, 10.18, 9.88 [14:16:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.60, 3.90, 3.72 [14:16:28] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.55, 3.74, 3.71 [14:20:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.30, 10.56, 10.07 [14:20:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.03, 3.95, 3.81 [14:21:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.15, 10.34, 10.20 [14:22:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.60, 8.88, 9.53 [14:22:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.10, 3.66, 3.72 [14:23:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.85, 9.19, 9.81 [14:26:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.32, 4.14, 3.85 [14:26:24] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.85, 4.01, 3.83 [14:27:06] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.93, 3.45, 3.08 [14:27:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.35, 3.56, 1.98 [14:27:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.66, 10.62, 10.26 [14:28:23] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.85, 3.63, 3.71 [14:29:05] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.25, 3.26, 3.05 [14:29:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.59, 3.21, 2.05 [14:30:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.67, 3.89, 3.84 [14:32:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.21, 3.87, 3.78 [14:33:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.53, 11.35, 10.58 [14:34:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.34, 3.61, 3.69 [14:39:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.49, 3.85, 3.39 [14:41:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.12, 3.58, 3.35 [14:41:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.30, 11.98, 11.39 [14:41:52] PROBLEM - mwtask141 Disk Space on mwtask141 is WARNING: DISK WARNING - free space: / 7708 MB (10% inode=90%); [14:43:00] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.47, 3.22, 3.24 [14:43:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.34, 11.57, 9.83 [14:45:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.47, 11.09, 9.84 [14:46:16] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.55, 4.09, 3.84 [14:46:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.31, 3.97, 3.74 [14:48:15] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.46, 3.85, 3.78 [14:48:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.80, 3.51, 3.59 [14:49:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.02, 11.19, 10.97 [14:51:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.40, 10.57, 10.76 [14:51:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.92, 3.67, 3.43 [14:54:13] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.41, 3.64, 3.67 [14:54:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.45, 9.47, 9.72 [14:55:37] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.65, 3.18, 3.29 [14:56:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.35, 3.44, 3.59 [14:57:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.85, 2.96, 1.90 [14:59:30] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.15, 2.51, 1.85 [14:59:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.05, 9.27, 10.07 [15:02:22] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.94, 3.25, 3.40 [15:06:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.62, 3.40, 3.43 [15:06:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.54, 11.63, 10.73 [15:08:22] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.26, 3.27, 3.37 [15:10:11] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.54, 2.92, 3.33 [15:10:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.76, 11.21, 10.83 [15:14:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.43, 3.37, 3.42 [15:16:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.85, 11.02, 10.72 [15:16:11] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.56, 3.22, 3.37 [15:17:14] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.78, 3.76, 3.55 [15:19:12] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.59, 3.86, 3.59 [15:20:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.43, 3.86, 3.60 [15:21:10] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.04, 3.45, 3.47 [15:22:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.77, 3.71, 3.57 [15:23:08] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.67, 3.73, 3.56 [15:24:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.92, 11.46, 10.36 [15:26:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.70, 4.24, 3.81 [15:26:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.67, 11.81, 10.64 [15:28:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.31, 3.88, 3.73 [15:29:02] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.84, 3.95, 3.72 [15:29:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.78, 11.98, 11.71 [15:31:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.41, 4.08, 3.79 [15:32:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.02, 12.03, 11.01 [15:32:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.30, 3.77, 3.71 [15:33:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.08, 11.55, 11.49 [15:34:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.59, 10.97, 10.74 [15:34:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.14, 3.73, 3.69 [15:36:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.11, 3.68, 3.63 [15:36:54] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.20, 3.76, 3.73 [15:38:53] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.62, 3.98, 3.80 [15:40:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.29, 3.77, 3.75 [15:43:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.03, 10.01, 11.34 [15:44:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.39, 3.99, 3.82 [15:46:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 3.95, 4.08, 3.88 [15:48:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.94, 3.96, 3.86 [15:48:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.11, 10.28, 10.45 [15:52:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.17, 4.17, 3.92 [15:52:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.34, 11.46, 10.94 [15:52:39] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.08, 3.72, 3.74 [15:54:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.63, 3.56, 3.67 [15:56:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.28, 3.91, 3.89 [15:59:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.38, 9.69, 10.19 [16:00:31] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.14, 3.65, 3.67 [16:02:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.83, 3.92, 3.84 [16:02:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.33, 9.01, 10.02 [16:04:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.93, 3.84, 3.81 [16:04:27] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.82, 3.81, 3.73 [16:05:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.25, 11.67, 10.73 [16:06:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.00, 4.56, 4.09 [16:06:25] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 6.56, 4.75, 4.08 [16:07:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.40, 10.91, 10.56 [16:09:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.60, 11.65, 10.83 [16:10:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.35, 4.42, 2.46 [16:10:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.14, 3.79, 3.89 [16:10:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.05, 11.47, 10.50 [16:10:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.17, 3.94, 3.93 [16:12:16] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.562 second response time [16:13:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.31, 12.00, 11.25 [16:14:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.86, 3.98, 2.83 [16:16:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.92, 2.99, 2.60 [16:17:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.35, 12.13, 11.42 [16:19:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.35, 11.25, 11.18 [16:20:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.82, 10.66, 11.04 [16:25:12] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.36, 3.54, 3.23 [16:27:11] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.72, 3.20, 3.14 [16:28:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.33, 9.16, 10.18 [16:28:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.73, 3.78, 3.94 [16:30:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.16, 3.94, 3.99 [16:31:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.49, 10.66, 10.60 [16:32:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.50, 11.37, 10.87 [16:32:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.76, 3.58, 3.85 [16:33:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.91, 10.92, 10.70 [16:34:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 10.88, 10.76 [16:34:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.11, 3.73, 3.87 [16:35:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.85, 12.09, 11.19 [16:36:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.06, 3.51, 3.93 [16:36:14] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.46, 3.46, 3.08 [16:36:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.64, 3.67, 3.83 [16:38:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.59, 3.88, 4.01 [16:38:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.68, 9.10, 10.05 [16:38:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.66, 3.32, 3.08 [16:38:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.20, 3.93, 3.91 [16:40:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.14, 3.60, 3.89 [16:41:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.42, 10.44, 10.94 [16:42:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.20, 3.80, 3.87 [16:44:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.96, 4.01, 3.95 [16:46:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.68, 3.68, 3.82 [16:48:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.24, 3.77, 3.83 [16:50:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.10, 3.99, 3.93 [16:50:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.55, 10.35, 10.11 [16:52:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.68, 3.86, 3.90 [16:52:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.96, 9.57, 9.85 [16:53:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.23, 10.14, 10.25 [16:54:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.86, 4.38, 4.09 [16:55:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.47, 9.74, 10.08 [16:56:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.73, 10.47, 10.18 [16:56:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.34, 3.95, 3.94 [16:58:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.15, 9.69, 9.93 [17:00:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.12, 4.23, 4.02 [17:01:14] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/a4484ceb41e3...77936eba1429 [17:01:15] [url] Comparing a4484ceb41e3...77936eba1429 · miraheze/ErrorPages · GitHub | github.com [17:01:16] [02miraheze/ErrorPages] 07Universal-Omega 0377936eb - databaseMaintenance: add more information [17:01:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.19, 9.91, 9.92 [17:01:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.80, 3.76, 3.41 [17:02:10] miraheze/ErrorPages - Universal-Omega the build passed. [17:02:37] !log [universalomega@mwtask141] starting deploy of {'pull': 'errorpages', 'errorpages': True} to all [17:02:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:47] !log [universalomega@mwtask141] finished deploy of {'pull': 'errorpages', 'errorpages': True} to all - SUCCESS in 9s [17:02:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.31, 11.10, 10.35 [17:04:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.89, 3.62, 3.84 [17:04:24] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/77936eba1429...1a2e94d06561 [17:04:24] [url] Comparing 77936eba1429...1a2e94d06561 · miraheze/ErrorPages · GitHub | github.com [17:04:27] [02miraheze/ErrorPages] 07Universal-Omega 031a2e94d - databaseMaintenance: use small tags [17:04:30] !log [universalomega@mwtask141] starting deploy of {'pull': 'errorpages', 'errorpages': True} to all [17:04:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:37] !log [universalomega@mwtask141] finished deploy of {'pull': 'errorpages', 'errorpages': True} to all - SUCCESS in 6s [17:04:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1243162751 secs [17:05:15] miraheze/ErrorPages - Universal-Omega the build passed. [17:05:27] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/1a2e94d06561...abc680ac3909 [17:05:27] [url] Comparing 1a2e94d06561...abc680ac3909 · miraheze/ErrorPages · GitHub | github.com [17:05:30] [02miraheze/ErrorPages] 07Universal-Omega 03abc680a - databaseMaintenance: add small to CSS [17:05:31] !log [universalomega@mwtask141] starting deploy of {'pull': 'errorpages', 'errorpages': True} to all [17:05:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:37] !log [universalomega@mwtask141] finished deploy of {'pull': 'errorpages', 'errorpages': True} to all - SUCCESS in 6s [17:05:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.02, 11.21, 10.50 [17:05:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.98, 3.60, 3.88 [17:06:27] miraheze/ErrorPages - Universal-Omega the build passed. [17:06:30] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/abc680ac3909...c93232be7138 [17:06:30] [url] Comparing abc680ac3909...c93232be7138 · miraheze/ErrorPages · GitHub | github.com [17:06:31] [02miraheze/ErrorPages] 07Universal-Omega 03c93232b - Add whitespace [17:06:36] !log [universalomega@mwtask141] starting deploy of {'pull': 'errorpages', 'errorpages': True} to all [17:06:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:45] !log [universalomega@mwtask141] finished deploy of {'pull': 'errorpages', 'errorpages': True} to all - SUCCESS in 9s [17:06:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.06066310406 secs [17:07:30] miraheze/ErrorPages - Universal-Omega the build passed. [17:07:35] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/c93232be7138...a585eeb19d90 [17:07:36] [url] Comparing c93232be7138...a585eeb19d90 · miraheze/ErrorPages · GitHub | github.com [17:07:38] [02miraheze/ErrorPages] 07Universal-Omega 03a585eeb - Fix [17:07:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.39, 12.15, 10.93 [17:07:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.47, 3.60, 3.31 [17:08:04] !log [universalomega@mwtask141] starting deploy of {'pull': 'errorpages', 'errorpages': True} to all [17:08:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:08:10] !log [universalomega@mwtask141] finished deploy of {'pull': 'errorpages', 'errorpages': True} to all - SUCCESS in 6s [17:08:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:08:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.49, 4.02, 3.92 [17:08:26] miraheze/ErrorPages - Universal-Omega the build passed. [17:09:21] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/a585eeb19d90...6e7e68eb4bfb [17:09:22] [url] Comparing a585eeb19d90...6e7e68eb4bfb · miraheze/ErrorPages · GitHub | github.com [17:09:22] [02miraheze/ErrorPages] 07Universal-Omega 036e7e68e - Re-order [17:09:24] !log [universalomega@mwtask141] starting deploy of {'pull': 'errorpages', 'errorpages': True} to all [17:09:31] !log [universalomega@mwtask141] finished deploy of {'pull': 'errorpages', 'errorpages': True} to all - SUCCESS in 6s [17:09:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:09:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:09:40] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.17, 3.76, 3.40 [17:09:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.19, 11.18, 10.72 [17:10:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.18, 3.85, 3.88 [17:10:16] miraheze/ErrorPages - Universal-Omega the build passed. [17:10:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.54, 3.73, 3.83 [17:11:36] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.50, 3.31, 3.28 [17:12:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.57, 3.65, 3.81 [17:12:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.42, 10.81, 10.12 [17:13:49] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.06, 3.37, 3.40 [17:14:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.76, 4.53, 4.10 [17:14:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/654d1e414a7e [17:14:19] [02miraheze/mw-config] 07Universal-Omega 03654d1e4 - Enable `$wgDatabaseClustersMaintenance` for `c5` [17:14:20] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [17:14:20] [url] Page not found · GitHub · GitHub | github.com [17:14:23] [02mw-config] 07Universal-Omega opened pull request 03#5023: Enable `$wgDatabaseClustersMaintenance` for `c5` - 13https://github.com/miraheze/mw-config/pull/5023 [17:14:23] [url] Page not found · GitHub · GitHub | github.com [17:15:24] miraheze/mw-config - Universal-Omega the build passed. [17:16:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.10, 3.74, 3.76 [17:17:46] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.18, 3.50, 3.46 [17:18:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.40, 3.91, 3.95 [17:18:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.38, 11.84, 10.91 [17:18:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.20, 3.55, 3.69 [17:23:43] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.64, 3.60, 3.47 [17:24:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.38, 3.73, 3.70 [17:25:42] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.36, 3.73, 3.55 [17:26:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.92, 3.73, 3.70 [17:27:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 10.88, 10.54 [17:29:41] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.05, 3.04, 3.33 [17:29:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.13, 10.56, 10.47 [17:33:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.94, 3.39, 3.44 [17:33:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.89, 12.64, 11.33 [17:34:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.28, 4.11, 3.75 [17:35:37] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.66, 3.20, 3.37 [17:38:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.79, 3.94, 3.75 [17:38:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.57, 8.42, 9.81 [17:39:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.03, 11.58, 11.38 [17:40:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.19, 3.67, 3.58 [17:41:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 12.13, 11.63 [17:42:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.27, 10.41, 10.27 [17:42:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.36, 3.43, 3.43 [17:43:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.17, 11.23, 11.35 [17:44:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.26, 10.56, 10.33 [17:46:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.32, 4.19, 3.92 [17:46:31] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.30, 3.12, 3.28 [17:48:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.44, 3.79, 3.80 [17:48:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.41, 10.07, 10.19 [17:48:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.46, 3.95, 3.81 [17:52:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.72, 3.96, 3.84 [17:52:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.57, 4.30, 3.98 [17:56:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.26, 3.84, 3.87 [17:56:56] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset -0.1088261604 secs [17:57:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.80, 8.47, 9.93 [17:58:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.16, 3.85, 3.87 [18:06:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.19, 3.73, 3.72 [18:06:22] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.62, 2.87, 3.35 [18:08:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.35, 3.75, 3.74 [18:10:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.25, 3.78, 3.74 [18:12:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.92, 3.80, 3.75 [18:14:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 6.32, 4.60, 4.04 [18:14:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9b884f32d37b [18:14:20] [02miraheze/mw-config] 07Universal-Omega 039b884f3 - Temporarily use mem141 for all caching [18:14:23] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [18:14:23] [url] Page not found · GitHub · GitHub | github.com [18:14:24] [02mw-config] 07Universal-Omega opened pull request 03#5024: Temporarily use mem141 for all caching - 13https://github.com/miraheze/mw-config/pull/5024 [18:14:24] [url] Page not found · GitHub · GitHub | github.com [18:15:35] miraheze/mw-config - Universal-Omega the build passed. [18:15:44] [02mw-config] 07Universal-Omega closed pull request 03#5024: Temporarily use mem141 for all caching - 13https://github.com/miraheze/mw-config/pull/5024 [18:15:44] [url] Page not found · GitHub · GitHub | github.com [18:15:47] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [18:15:48] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [18:15:48] ... [18:15:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6575a3485dc2...a568203efb6d [18:15:49] [url] Comparing 6575a3485dc2...a568203efb6d · miraheze/mw-config · GitHub | github.com [18:15:51] [02miraheze/mw-config] 07Universal-Omega 03a568203 - Temporarily use mem141 for all caching (#5024) [18:16:10] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [18:16:18] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [18:16:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:16:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:16:47] miraheze/mw-config - Universal-Omega the build passed. [18:16:56] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.0876776576 secs [18:17:12] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.94, 3.36, 3.42 [18:19:10] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.83, 3.13, 3.32 [18:21:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.76, 10.16, 9.19 [18:22:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.23, 3.83, 3.90 [18:23:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.97, 11.65, 9.84 [18:26:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.29, 10.07, 8.68 [18:27:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.57, 3.26, 3.28 [18:27:40] !log [@test131] starting deploy of {'config': True} to all [18:27:41] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:27:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.06, 11.35, 10.16 [18:28:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.32, 9.54, 8.65 [18:28:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.08, 3.56, 3.38 [18:30:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.62, 3.56, 3.40 [18:31:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 25.62, 15.94, 12.04 [18:32:04] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:32:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 33.23, 19.38, 12.60 [18:33:34] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:33:37] PROBLEM - cloud12 Current Load on cloud12 is CRITICAL: CRITICAL - load average: 114.08, 65.50, 35.76 [18:34:02] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 20.92, 17.29, 10.55 [18:34:52] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.82, 3.18, 3.29 [18:35:33] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.258 second response time [18:35:35] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 2 backends are down. mw121 mw141 [18:35:52] PROBLEM - mwtask141 Disk Space on mwtask141 is CRITICAL: DISK CRITICAL - free space: / 4236 MB (5% inode=90%); [18:37:35] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [18:37:37] PROBLEM - cloud12 Current Load on cloud12 is WARNING: WARNING - load average: 22.36, 44.39, 34.34 [18:38:03] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 4.70, 10.26, 9.18 [18:38:16] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.917 second response time [18:39:37] RECOVERY - cloud12 Current Load on cloud12 is OK: OK - load average: 15.84, 34.69, 32.01 [18:40:02] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 3.34, 7.93, 8.45 [18:40:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.23, 3.52, 3.58 [18:45:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.15, 10.09, 11.85 [18:46:36] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.68, 3.56, 3.38 [18:48:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.78, 3.92, 3.81 [18:48:34] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.03, 3.51, 3.37 [18:48:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.47, 3.45, 3.18 [18:49:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.73, 11.50, 11.82 [18:50:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.83, 9.52, 11.39 [18:50:32] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.23, 3.08, 3.23 [18:50:45] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.51, 3.21, 3.13 [18:51:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.97, 3.20, 2.96 [18:52:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.53, 11.46, 11.84 [18:53:00] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.00, 3.21, 3.00 [18:56:25] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.20, 3.75, 3.46 [18:56:59] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.17, 3.49, 3.17 [18:57:44] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.57, 2.68, 1.79 [18:58:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.98, 4.40, 3.98 [18:58:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.89, 3.77, 3.50 [18:58:58] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.90, 3.34, 3.16 [18:59:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.02, 11.37, 11.73 [18:59:44] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.24, 4.26, 2.51 [19:00:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.94, 10.83, 11.66 [19:01:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.01, 3.41, 2.41 [19:02:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.80, 3.90, 3.86 [19:02:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.78, 4.07, 3.67 [19:03:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.70, 2.46, 2.18 [19:05:17] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.85, 3.76, 3.37 [19:05:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.73, 12.83, 12.17 [19:06:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.45, 4.00, 3.74 [19:09:57] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.06, 10.47, 8.16 [19:10:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.77, 4.24, 3.93 [19:11:05] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 6.43, 4.46, 3.69 [19:11:56] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.11, 8.62, 7.77 [19:13:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.03, 10.55, 11.60 [19:14:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.16, 3.87, 3.77 [19:16:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.93, 3.47, 3.49 [19:18:49] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.04, 3.24, 3.40 [19:23:28] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 11695 MB (16% inode=90%); [19:24:12] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.44, 9.14, 6.97 [19:24:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.19, 8.97, 10.13 [19:26:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:26:12] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.22, 7.75, 6.71 [19:27:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.05, 10.73, 11.02 [19:31:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 30.16, 18.12, 13.28 [19:31:37] PROBLEM - cloud12 Current Load on cloud12 is CRITICAL: CRITICAL - load average: 114.21, 65.65, 36.20 [19:31:38] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 27.37, 16.21, 10.31 [19:31:49] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.81, 3.66, 3.90 [19:33:18] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw142 [19:33:35] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw142 [19:34:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.07, 3.06, 3.80 [19:34:32] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw142 [19:35:34] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.52, 9.89, 9.05 [19:36:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:36:16] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.69, 3.37, 3.30 [19:38:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.367 second response time [19:38:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.39, 3.13, 3.23 [19:38:22] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.270 second response time [19:38:24] PROBLEM - mw121 Disk Space on mw121 is WARNING: DISK WARNING - free space: / 1532 MB (6% inode=81%); [19:39:43] PROBLEM - mw122 Disk Space on mw122 is WARNING: DISK WARNING - free space: / 2007 MB (9% inode=81%); [19:40:04] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw121 [19:40:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.96, 3.48, 2.55 [19:41:35] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [19:42:03] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [19:42:32] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [19:42:46] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.70, 2.84, 2.43 [19:43:19] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [19:44:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.11, 3.63, 3.72 [19:46:13] PROBLEM - mw121 Disk Space on mw121 is CRITICAL: DISK CRITICAL - free space: / 1308 MB (5% inode=81%); [19:47:32] PROBLEM - mw122 Disk Space on mw122 is CRITICAL: DISK CRITICAL - free space: / 238 MB (1% inode=81%); [19:48:11] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.43, 3.80, 3.79 [19:48:33] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:35] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 21.01, 15.82, 11.29 [19:49:57] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw122 [19:50:16] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.998 second response time [19:50:32] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw122 [19:51:19] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw142 [19:51:35] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw122 [19:52:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.06, 3.79, 3.77 [19:52:32] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.388 second response time [19:53:32] PROBLEM - mw142 Disk Space on mw142 is CRITICAL: DISK CRITICAL - free space: / 701 MB (2% inode=82%); [19:53:35] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [19:53:50] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5796 bytes in 0.008 second response time [19:53:57] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [19:54:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.52, 3.56, 3.68 [19:54:39] PROBLEM - cloud14 Current Load on cloud14 is WARNING: WARNING - load average: 76.20, 63.76, 40.25 [19:55:53] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.786 second response time [19:56:02] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 33.39, 17.84, 10.78 [19:56:12] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.43, 9.77, 7.82 [19:56:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.46, 3.63, 3.67 [19:56:39] RECOVERY - cloud14 Current Load on cloud14 is OK: OK - load average: 49.41, 60.25, 41.97 [19:57:06] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.04, 10.35, 8.10 [19:57:18] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [19:58:12] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.38, 8.42, 7.55 [19:58:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.33, 3.49, 3.62 [19:58:27] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.78, 3.49, 3.12 [19:59:06] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.78, 8.99, 7.87 [19:59:37] PROBLEM - cloud12 Current Load on cloud12 is WARNING: WARNING - load average: 16.35, 28.78, 46.67 [20:00:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.40, 4.12, 3.78 [20:00:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.15, 3.74, 3.69 [20:00:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 5.13, 3.77, 3.25 [20:00:32] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [20:02:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.75, 3.88, 3.76 [20:02:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.51, 3.59, 3.24 [20:03:37] RECOVERY - cloud12 Current Load on cloud12 is OK: OK - load average: 17.55, 22.41, 39.93 [20:04:02] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 4.34, 9.94, 10.47 [20:04:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.39, 3.89, 3.79 [20:04:25] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 3.22, 3.37, 3.20 [20:06:03] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.70, 8.37, 9.84 [20:06:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.02, 4.22, 3.91 [20:06:46] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.04, 3.69, 3.35 [20:07:34] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 4.83, 7.53, 11.17 [20:10:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.30, 3.89, 3.86 [20:10:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.28, 8.71, 11.84 [20:10:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.20, 3.94, 3.80 [20:10:37] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.68, 3.23, 3.25 [20:11:34] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.15, 5.90, 9.63 [20:12:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.85, 3.57, 3.69 [20:14:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.40, 3.91, 3.86 [20:16:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.30, 3.73, 3.80 [20:16:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.29, 4.07, 3.82 [20:17:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.42, 10.06, 11.87 [20:18:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.92, 3.74, 3.74 [20:21:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.79, 11.32, 11.97 [20:22:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.67, 4.25, 3.93 [20:23:15] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 5.66, 3.93, 3.33 [20:24:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.51, 3.95, 3.86 [20:25:14] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.70, 3.39, 3.20 [20:25:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.40, 11.19, 11.90 [20:28:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.38, 8.22, 9.77 [20:30:12] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 3.02, 3.09, 3.40 [20:32:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1440839469 secs [20:32:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.53, 3.94, 3.85 [20:32:31] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.67, 2.85, 1.75 [20:33:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.69, 7.72, 9.92 [20:34:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.48, 3.76, 3.80 [20:34:26] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.58, 2.98, 1.95 [20:36:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.24, 4.20, 3.75 [20:36:14] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.91, 3.61, 3.12 [20:37:27] PROBLEM - mw122 Disk Space on mw122 is WARNING: DISK WARNING - free space: / 2121 MB (9% inode=81%); [20:38:08] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07877501845 secs [20:38:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.71, 3.95, 3.71 [20:38:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.94, 3.23, 3.03 [20:40:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.93, 3.95, 2.47 [20:40:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.33, 3.96, 3.73 [20:40:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.62, 3.51, 3.63 [20:42:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.99, 2.85, 2.25 [20:42:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.43, 3.68, 3.65 [20:42:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.12, 3.30, 3.53 [20:44:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.07, 4.02, 3.79 [20:44:13] PROBLEM - mw121 Disk Space on mw121 is WARNING: DISK WARNING - free space: / 2372 MB (10% inode=81%); [20:46:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.58, 9.56, 9.12 [20:46:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.39, 3.76, 3.64 [20:48:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 10.92, 9.70 [20:48:14] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.78, 3.09, 1.95 [20:48:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.89, 3.45, 3.55 [20:49:57] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/c5c13d2ccef1 [20:50:00] [02miraheze/dns] 07paladox 03c5c13d2 - Depool swiftproxy131 [20:50:02] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [20:50:02] [url] Page not found · GitHub · GitHub | github.com [20:50:05] [02dns] 07paladox opened pull request 03#383: Depool swiftproxy131 - 13https://github.com/miraheze/dns/pull/383 [20:50:05] ... [20:50:06] [02dns] 07paladox closed pull request 03#383: Depool swiftproxy131 - 13https://github.com/miraheze/dns/pull/383 [20:50:06] ... [20:50:09] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/963a495cdcd5...ae7b0e8a6b62 [20:50:09] [url] Comparing 963a495cdcd5...ae7b0e8a6b62 · miraheze/dns · GitHub | github.com [20:50:10] [02miraheze/dns] 07paladox 03ae7b0e8 - Depool swiftproxy131 (#383) [20:50:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.19, 3.69, 3.73 [20:50:12] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [20:50:12] [url] Page not found · GitHub · GitHub | github.com [20:50:14] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [20:50:14] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 3.10, 3.35, 2.20 [20:51:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/7bde99971c5f [20:51:14] [02miraheze/mw-config] 07paladox 037bde999 - Set c2 and c5 into read only [20:51:17] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [20:51:18] [url] Page not found · GitHub · GitHub | github.com [20:51:20] [02mw-config] 07paladox opened pull request 03#5025: Set c2 and c5 into read only - 13https://github.com/miraheze/mw-config/pull/5025 [20:51:21] [url] Page not found · GitHub · GitHub | github.com [20:52:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.34, 3.78, 3.75 [20:52:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.38, 3.68, 3.60 [20:52:22] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.15, 1.72, 1.01 [20:52:24] miraheze/mw-config - paladox the build passed. [20:54:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.53, 3.73, 3.74 [20:54:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.59, 11.21, 10.41 [20:54:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.52, 3.65, 3.60 [20:55:31] PROBLEM - phab121 Disk Space on phab121 is WARNING: DISK WARNING - free space: / 2103 MB (7% inode=84%); [20:56:14] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.58, 3.50, 3.23 [20:58:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.06, 9.41, 9.85 [20:58:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.87, 3.16, 3.12 [20:59:31] PROBLEM - phab121 Disk Space on phab121 is CRITICAL: DISK CRITICAL - free space: / 1331 MB (5% inode=84%); [21:00:05] [02mw-config] 07Universal-Omega closed pull request 03#5023: Enable `$wgDatabaseClustersMaintenance` for `c5` - 13https://github.com/miraheze/mw-config/pull/5023 [21:00:06] [url] Page not found · GitHub · GitHub | github.com [21:00:07] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [21:00:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a568203efb6d...5051518106bd [21:00:10] [url] Comparing a568203efb6d...5051518106bd · miraheze/mw-config · GitHub | github.com [21:00:11] [02miraheze/mw-config] 07Universal-Omega 035051518 - Enable `$wgDatabaseClustersMaintenance` for `c5` (#5023) [21:00:14] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [21:00:15] [url] Page not found · GitHub · GitHub | github.com [21:00:15] [02mw-config] 07Universal-Omega closed pull request 03#5025: Set c2 and c5 into read only - 13https://github.com/miraheze/mw-config/pull/5025 [21:00:16] ... [21:00:18] [02miraheze/mw-config] 07Universal-Omega deleted branch 03paladox-patch-1 [21:00:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5051518106bd...2028860123c9 [21:00:20] [url] Comparing 5051518106bd...2028860123c9 · miraheze/mw-config · GitHub | github.com [21:00:21] [02miraheze/mw-config] 07paladox 032028860 - Set c2 and c5 into read only (#5025) [21:00:22] [02mw-config] 07Universal-Omega deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:00:23] ... [21:00:24] !log set db101 into read only [21:00:34] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [21:00:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:58] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [21:01:13] miraheze/mw-config - Universal-Omega the build passed. [21:01:59] !log shutdown all VMs on cloud13 [21:02:13] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.258 second response time [21:02:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.73, 10.64, 10.42 [21:02:52] !log dump meta, login, incidents and mhglobal on db141 [21:03:01] [Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:03:48] PROBLEM - mw132 Puppet on mw132 is CRITICAL: connect to address 2a10:6740::6:404 port 5666: No route to hostconnect to host 2a10:6740::6:404 port 5666: No route to host [21:03:50] PROBLEM - matomo131 SSH on matomo131 is CRITICAL: connect to address 2a10:6740::6:407 and port 22: No route to host [21:03:50] PROBLEM - db131 Puppet on db131 is CRITICAL: connect to address 2a10:6740::6:408 port 5666: No route to hostconnect to host 2a10:6740::6:408 port 5666: No route to host [21:03:50] PROBLEM - db131 Current Load on db131 is CRITICAL: connect to address 2a10:6740::6:408 port 5666: No route to hostconnect to host 2a10:6740::6:408 port 5666: No route to host [21:03:50] PROBLEM - db131 PowerDNS Recursor on db131 is CRITICAL: connect to address 2a10:6740::6:408 port 5666: No route to hostconnect to host 2a10:6740::6:408 port 5666: No route to host [21:03:54] PROBLEM - mw132 JobRunner Service on mw132 is CRITICAL: connect to address 2a10:6740::6:404 port 5666: No route to hostconnect to host 2a10:6740::6:404 port 5666: No route to host [21:03:55] PROBLEM - matomo131 Disk Space on matomo131 is CRITICAL: connect to address 2a10:6740::6:407 port 5666: No route to hostconnect to host 2a10:6740::6:407 port 5666: No route to host [21:03:55] PROBLEM - Host prometheus131 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:402) [21:03:56] PROBLEM - mw131 SSH on mw131 is CRITICAL: connect to address 2a10:6740::6:403 and port 22: No route to host [21:03:57] PROBLEM - mem131 Disk Space on mem131 is CRITICAL: connect to address 2a10:6740::6:405 port 5666: No route to hostconnect to host 2a10:6740::6:405 port 5666: No route to host [21:03:57] PROBLEM - ping6 on es131 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:401) [21:03:59] PROBLEM - mw132 SSH on mw132 is CRITICAL: connect to address 2a10:6740::6:404 and port 22: No route to host [21:04:00] PROBLEM - test131 HTTPS on test131 is CRITICAL: connect to address 2a10:6740::6:406 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [21:04:00] PROBLEM - test131 NTP time on test131 is CRITICAL: connect to address 2a10:6740::6:406 port 5666: No route to hostconnect to host 2a10:6740::6:406 port 5666: No route to host [21:04:02] PROBLEM - Host matomo131 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:407) [21:04:05] PROBLEM - es131 SSH on es131 is CRITICAL: connect to address 2a10:6740::6:401 and port 22: No route to host [21:04:06] PROBLEM - test131 conntrack_table_size on test131 is CRITICAL: connect to address 2a10:6740::6:406 port 5666: No route to hostconnect to host 2a10:6740::6:406 port 5666: No route to host [21:04:06] PROBLEM - test131 php-fpm on test131 is CRITICAL: connect to address 2a10:6740::6:406 port 5666: No route to hostconnect to host 2a10:6740::6:406 port 5666: No route to host [21:04:06] PROBLEM - test131 PowerDNS Recursor on test131 is CRITICAL: connect to address 2a10:6740::6:406 port 5666: No route to hostconnect to host 2a10:6740::6:406 port 5666: No route to host [21:04:06] PROBLEM - test131 nutcracker process on test131 is CRITICAL: connect to address 2a10:6740::6:406 port 5666: No route to hostconnect to host 2a10:6740::6:406 port 5666: No route to host [21:04:10] PROBLEM - db131 conntrack_table_size on db131 is CRITICAL: connect to address 2a10:6740::6:408 port 5666: No route to hostconnect to host 2a10:6740::6:408 port 5666: No route to host [21:04:11] gonna quite icinga CosmicAlpha [21:04:14] PROBLEM - mw131 nutcracker process on mw131 is CRITICAL: connect to address 2a10:6740::6:403 port 5666: No route to hostconnect to host 2a10:6740::6:403 port 5666: No route to host [21:04:14] PROBLEM - mw131 conntrack_table_size on mw131 is CRITICAL: connect to address 2a10:6740::6:403 port 5666: No route to hostconnect to host 2a10:6740::6:403 port 5666: No route to host [21:04:14] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: connect to address 2a10:6740::6:403 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [21:04:14] PROBLEM - ping6 on mw131 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:403) [21:04:14] PROBLEM - mw131 Current Load on mw131 is CRITICAL: connect to address 2a10:6740::6:403 port 5666: No route to hostconnect to host 2a10:6740::6:403 port 5666: No route to host [21:04:14] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: connect to address 2a10:6740::6:406 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket PROBLEM - test131 ferm_active on test131 is CRITICAL: connect to address 2a10:6740::6:406 port 5666: No route to hostconnect to host 2a10:6740::6:406 port 5666: No route to host [21:04:14] PROBLEM - test131 Current Load on test131 is CRITICAL: connect to address 2a10:6740::6:406 port 5666: No route to hostconnect to host 2a10:6740::6:406 port 5666: No route to host [21:04:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.20, 9.70, 10.09 [21:04:23] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: connect to address 2a10:6740::6:404 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [21:04:23] PROBLEM - mem131 conntrack_table_size on mem131 is CRITICAL: connect to address 2a10:6740::6:405 port 5666: No route to hostconnect to host 2a10:6740::6:405 port 5666: No route to host [21:04:23] PROBLEM - mem131 SSH on mem131 is CRITICAL: connect to address 2a10:6740::6:405 and port 22: No route to host [21:04:24] PROBLEM - mw131 php-fpm on mw131 is CRITICAL: connect to address 2a10:6740::6:403 port 5666: No route to hostconnect to host 2a10:6740::6:403 port 5666: No route to host [21:04:24] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: connect to address 2a10:6740::6:403 port 5666: No route to hostconnect to host 2a10:6740::6:403 port 5666: No route to host [21:04:27] Thanks paladox [21:04:27] PROBLEM - mw132 PowerDNS Recursor on mw132 is CRITICAL: connect to address 2a10:6740::6:404 port 5666: No route to hostconnect to host 2a10:6740::6:404 port 5666: No route to host [21:04:27] PROBLEM - mem131 Puppet on mem131 is CRITICAL: connect to address 2a10:6740::6:405 port 5666: No route to hostconnect to host 2a10:6740::6:405 port 5666: No route to host [21:04:30] PROBLEM - mw132 php-fpm on mw132 is CRITICAL: connect to address 2a10:6740::6:404 port 5666: No route to hostconnect to host 2a10:6740::6:404 port 5666: No route to host [21:04:35] PROBLEM - es131 Puppet on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: No route to hostconnect to host 2a10:6740::6:401 port 5666: No route to host [21:04:36] PROBLEM - mem131 ferm_active on mem131 is CRITICAL: connect to address 2a10:6740::6:405 port 5666: No route to hostconnect to host 2a10:6740::6:405 port 5666: No route to host [21:04:39] PROBLEM - test131 JobChron Service on test131 is CRITICAL: connect to address 2a10:6740::6:406 port 5666: No route to hostconnect to host 2a10:6740::6:406 port 5666: No route to host [21:04:39] PROBLEM - ping6 on test131 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:406) [21:04:41] PROBLEM - es131 Current Load on es131 is CRITICAL: connect to address 2a10:6740::6:401 port 5666: No route to hostconnect to host 2a10:6740::6:401 port 5666: No route to host [21:04:43] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: connect to address 2a10:6740::6:404 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [21:05:35] !log shutdown cloud13 [21:07:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/c98518851a45 [21:07:36] [02miraheze/mw-config] 07paladox 03c985188 - Unset global wide read only for db101 [21:07:38] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [21:07:39] [url] Page not found · GitHub · GitHub | github.com [21:07:39] [02mw-config] 07paladox opened pull request 03#5026: Unset global wide read only for db101 - 13https://github.com/miraheze/mw-config/pull/5026 [21:07:40] [url] Page not found · GitHub · GitHub | github.com [21:08:47] miraheze/mw-config - paladox the build passed. [21:08:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/2c17ec006178 [21:09:01] [02miraheze/mw-config] 07paladox 032c17ec0 - Change DB default to db141 [21:09:02] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/mw-config [21:09:03] [url] Page not found · GitHub · GitHub | github.com [21:09:04] [02mw-config] 07paladox opened pull request 03#5027: Change DB default to db141 - 13https://github.com/miraheze/mw-config/pull/5027 [21:09:05] [url] Page not found · GitHub · GitHub | github.com [21:10:11] miraheze/mw-config - paladox the build passed. [21:11:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/660f23b93210 [21:11:07] [02miraheze/mw-config] 07paladox 03660f23b - Set default read only for db101 [21:11:09] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [21:11:10] [url] Page not found · GitHub · GitHub | github.com [21:11:10] [02mw-config] 07paladox opened pull request 03#5028: Set default read only for db101 - 13https://github.com/miraheze/mw-config/pull/5028 [21:11:11] ... [21:11:13] [02mw-config] 07paladox closed pull request 03#5028: Set default read only for db101 - 13https://github.com/miraheze/mw-config/pull/5028 [21:11:13] ... [21:11:14] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [21:11:15] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [21:11:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2028860123c9...1dbf76180cf3 [21:11:16] [url] Comparing 2028860123c9...1dbf76180cf3 · miraheze/mw-config · GitHub | github.com [21:11:19] [02miraheze/mw-config] 07paladox 031dbf761 - Set default read only for db101 (#5028) [21:11:29] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [21:11:58] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw131.miraheze.org [21:12:03] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [21:12:15] miraheze/mw-config - paladox the build passed. [21:12:22] miraheze/mw-config - paladox the build passed. [21:12:53] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 50s [23:39:02] RECOVERY - Host mem131 is UP: PING OK - Packet loss = 0%, RTA = 6.03 ms [23:39:07] RECOVERY - Host swiftproxy131 is UP: PING OK - Packet loss = 0%, RTA = 4.93 ms [23:39:08] RECOVERY - mem131 SSH on mem131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:39:19] RECOVERY - mem131 NTP time on mem131 is OK: NTP OK: Offset 0.001603782177 secs [23:39:19] RECOVERY - mem131 memcached on mem131 is OK: TCP OK - 0.038 second response time on 2a10:6740::6:405 port 11211 [23:39:26] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.32, 0.20, 0.18 [23:39:30] RECOVERY - Host mw131 is UP: PING OK - Packet loss = 0%, RTA = 4.28 ms [23:39:33] RECOVERY - Host cloud13 is UP: PING OK - Packet loss = 0%, RTA = 4.84 ms [23:39:36] RECOVERY - cloud13 NTP time on cloud13 is OK: NTP OK: Offset -0.0003062188625 secs [23:39:40] RECOVERY - cloud13 conntrack_table_size on cloud13 is OK: OK: nf_conntrack is 0 % full [23:39:40] RECOVERY - mon141 IRCEcho on mon141 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [23:39:41] RECOVERY - Host mw132 is UP: PING OK - Packet loss = 0%, RTA = 4.99 ms [23:39:42] RECOVERY - mw131 PowerDNS Recursor on mw131 is OK: DNS OK: 0.187 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [23:39:43] RECOVERY - mw131 conntrack_table_size on mw131 is OK: OK: nf_conntrack is 0 % full [23:39:44] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:39:44] RECOVERY - mw131 Disk Space on mw131 is OK: DISK OK - free space: / 11893 MB (50% inode=83%); [23:39:48] RECOVERY - swiftproxy131 ferm_active on swiftproxy131 is OK: OK ferm input default policy is set [23:39:50] RECOVERY - Host db131 is UP: PING OK - Packet loss = 0%, RTA = 7.03 ms [23:39:50] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 0.11, 0.41, 0.24 [23:39:51] RECOVERY - mw132 php-fpm on mw132 is OK: PROCS OK: 21 processes with command name 'php-fpm7.4' [23:39:53] RECOVERY - db131 NTP time on db131 is OK: NTP OK: Offset -0.0003080070019 secs [23:39:55] RECOVERY - Host es131 is UP: PING OK - Packet loss = 0%, RTA = 2.51 ms [23:39:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.61, 4.98, 3.10 [23:39:59] RECOVERY - cloud13 SSH on cloud13 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:40:00] RECOVERY - ping6 on es131 is OK: PING OK - Packet loss = 0%, RTA = 2.74 ms [23:40:00] RECOVERY - ping6 on mem131 is OK: PING OK - Packet loss = 0%, RTA = 3.30 ms [23:40:01] RECOVERY - cloud13 ferm_active on cloud13 is OK: OK ferm input default policy is set [23:40:01] RECOVERY - db131 SSH on db131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:40:03] RECOVERY - mw131 NTP time on mw131 is OK: NTP OK: Offset 0.0009635090828 secs [23:40:04] RECOVERY - mw131 nutcracker process on mw131 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [23:40:06] RECOVERY - mw132 PowerDNS Recursor on mw132 is OK: DNS OK: 0.209 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [23:40:13] RECOVERY - Host matomo131 is UP: PING OK - Packet loss = 0%, RTA = 4.04 ms [23:40:14] RECOVERY - Host test131 is UP: PING OK - Packet loss = 0%, RTA = 3.05 ms [23:40:16] RECOVERY - test131 Redis Process on test131 is OK: PROCS OK: 1 process with args 'redis-server' [23:40:19] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:40:20] RECOVERY - ping6 on mw132 is OK: PING OK - Packet loss = 0%, RTA = 3.77 ms [23:40:20] RECOVERY - mw131 SSH on mw131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:40:21] RECOVERY - Host prometheus131 is UP: PING OK - Packet loss = 0%, RTA = 3.51 ms [23:40:23] RECOVERY - ping6 on mw131 is OK: PING OK - Packet loss = 0%, RTA = 4.17 ms [23:40:26] RECOVERY - mem131 ferm_active on mem131 is OK: OK ferm input default policy is set [23:40:37] RECOVERY - ping6 on cloud13 is OK: PING OK - Packet loss = 0%, RTA = 3.27 ms [23:40:39] PROBLEM - prometheus131 Puppet on prometheus131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:40:43] RECOVERY - es131 NTP time on es131 is OK: NTP OK: Offset 0.001194119453 secs [23:40:44] RECOVERY - test131 nutcracker process on test131 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [23:40:44] RECOVERY - mem131 Disk Space on mem131 is OK: DISK OK - free space: / 6827 MB (72% inode=86%); [23:40:44] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 0.49, 0.42, 0.19 [23:40:49] RECOVERY - mw132 SSH on mw132 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:40:49] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 0.04, 0.14, 0.10 [23:40:49] RECOVERY - matomo131 Disk Space on matomo131 is OK: DISK OK - free space: / 11862 MB (62% inode=90%); [23:40:49] RECOVERY - mw132 Disk Space on mw132 is OK: DISK OK - free space: / 11894 MB (50% inode=82%); [23:40:54] RECOVERY - test131 ferm_active on test131 is OK: OK ferm input default policy is set [23:40:59] RECOVERY - swiftproxy131 NTP time on swiftproxy131 is OK: NTP OK: Offset 0.003065377474 secs [23:40:59] RECOVERY - mw132 nutcracker process on mw132 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [23:41:04] RECOVERY - mem131 conntrack_table_size on mem131 is OK: OK: nf_conntrack is 0 % full [23:41:04] RECOVERY - test131 JobChron Service on test131 is OK: PROCS OK: 1 process with args 'redisJobChronService' [23:41:09] RECOVERY - mw131 ferm_active on mw131 is OK: OK ferm input default policy is set [23:41:14] RECOVERY - ping6 on test131 is OK: PING OK - Packet loss = 0%, RTA = 0.94 ms [23:41:14] RECOVERY - mw132 NTP time on mw132 is OK: NTP OK: Offset 0.004531294107 secs [23:41:14] RECOVERY - es131 SSH on es131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:41:24] RECOVERY - db131 PowerDNS Recursor on db131 is OK: DNS OK: 0.055 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [23:41:24] RECOVERY - mw132 conntrack_table_size on mw132 is OK: OK: nf_conntrack is 0 % full [23:41:24] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 0.27, 0.36, 0.19 [23:41:29] RECOVERY - db131 conntrack_table_size on db131 is OK: OK: nf_conntrack is 0 % full [23:41:29] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:29] RECOVERY - matomo131 SSH on matomo131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:41:29] RECOVERY - mw131 php-fpm on mw131 is OK: PROCS OK: 21 processes with command name 'php-fpm7.4' [23:41:34] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:34] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:36] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: connect to address 2a10:6740::6:503 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:41:37] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:37] PROBLEM - puppet141 puppetserver on puppet141 is CRITICAL: connect to address 2a10:6740::6:507 and port 8140: Connection refused [23:41:38] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:38] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:38] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:38] PROBLEM - phab121 Puppet on phab121 is WARNING: WARNING: Puppet last ran 2 hours ago [23:41:38] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 8.86, 7.59, 3.20 [23:41:39] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: connect to address 2a10:6740::6:503 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:41:39] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:40] RECOVERY - test131 PowerDNS Recursor on test131 is OK: DNS OK: 0.027 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [23:41:40] PROBLEM - mw142 Puppet on mw142 is WARNING: WARNING: Puppet last ran 1 hour ago [23:41:41] PROBLEM - mw141 NTP time on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:41:41] PROBLEM - ping6 on mw141 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:502) [23:41:42] PROBLEM - mw141 conntrack_table_size on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:41:42] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: connect to address 2a10:6740::6:502 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [23:41:54] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:54] PROBLEM - matomo131 Puppet on matomo131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:41:55] PROBLEM - db141 PowerDNS Recursor on db141 is CRITICAL: connect to address 2a10:6740::6:508 port 5666: No route to hostconnect to host 2a10:6740::6:508 port 5666: No route to host [23:41:55] PROBLEM - db141 APT on db141 is CRITICAL: connect to address 2a10:6740::6:508 port 5666: No route to hostconnect to host 2a10:6740::6:508 port 5666: No route to host [23:41:56] PROBLEM - ping6 on db141 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:508) [23:41:56] PROBLEM - mw141 SSH on mw141 is CRITICAL: connect to address 2a10:6740::6:502 and port 22: No route to host [23:41:59] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:00] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:00] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:03] PROBLEM - db141 SSH on db141 is CRITICAL: connect to address 2a10:6740::6:508 and port 22: No route to host [23:42:04] PROBLEM - mw142 nutcracker process on mw142 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 115 (nutcracker), command name 'nutcracker' [23:42:04] PROBLEM - mem141 Puppet on mem141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:04] RECOVERY - test131 php-fpm on test131 is OK: PROCS OK: 9 processes with command name 'php-fpm7.4' [23:42:04] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:42:04] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 7.58, 5.38, 3.26 [23:42:07] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:09] RECOVERY - test131 conntrack_table_size on test131 is OK: OK: nf_conntrack is 0 % full [23:42:09] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:10] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:12] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5796 bytes in 0.014 second response time [23:42:14] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:14] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset 0.003051310778 secs [23:42:14] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [23:42:15] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:15] PROBLEM - mw141 Current Load on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:42:18] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:21] PROBLEM - Host db141 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:508) [23:42:21] PROBLEM - mw141 PowerDNS Recursor on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:42:21] PROBLEM - mw141 JobRunner Service on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:42:24] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:27] PROBLEM - swiftobject122 Puppet on swiftobject122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:27] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:31] PROBLEM - mw142 php-fpm on mw142 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4' [23:42:34] PROBLEM - mw141 Disk Space on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:42:35] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:36] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:42:39] PROBLEM - Host mw141 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:502) [23:42:41] PROBLEM - mw142 PowerDNS Recursor on mw142 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:43:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.68, 9.45, 5.29 [23:44:01] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [23:44:13] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.398 second response time [23:45:37] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 0.56, 3.66, 2.56 [23:46:03] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:47:37] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.33, 2.57, 2.30 [23:50:38] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:41] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.018 second response time [23:59:02] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds