[00:00:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 10.23, 9.45 [00:01:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.40, 11.73, 10.46 [00:02:18] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.24, 9.79, 9.38 [00:02:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.84, 2.96, 3.22 [00:03:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.44, 11.04, 10.35 [00:05:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.31, 9.94, 10.05 [00:06:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.89, 4.17, 3.63 [00:08:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.84, 3.79, 3.57 [00:10:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.64, 3.11, 3.35 [00:29:22] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.57, 3.15, 2.72 [00:33:22] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.77, 3.06, 2.80 [00:47:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.36, 10.07, 9.14 [00:47:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.96, 10.33, 9.37 [00:49:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.59, 10.11, 9.28 [00:49:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.42, 11.71, 10.02 [00:51:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.41, 11.41, 10.11 [00:53:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.14, 10.19, 9.82 [01:01:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.24, 3.29, 2.62 [01:03:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.07, 2.82, 2.53 [01:06:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.07, 10.44, 10.03 [01:08:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.86, 9.64, 9.79 [01:24:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 20.29, 8.34, 4.26 [01:32:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.39, 3.97, 3.91 [01:36:18] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.23, 2.52, 3.33 [01:38:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.56, 4.33, 3.53 [01:40:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.76, 3.43, 3.30 [01:42:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 2.99, 3.15 [01:51:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.05, 9.62, 8.70 [01:53:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.70, 10.27, 9.07 [01:53:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.64, 9.95, 8.94 [01:55:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.99, 9.35, 8.92 [01:55:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.04, 8.76, 8.62 [01:55:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 2.86, 4.19, 3.44 [01:57:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.34, 3.14, 3.14 [02:05:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.53, 11.11, 9.77 [02:06:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 30.86, 13.51, 6.58 [02:07:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.20, 9.99, 9.53 [02:20:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.92, 2.80, 3.88 [02:24:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.73, 2.19, 3.38 [02:25:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.18, 4.11, 2.96 [02:26:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.55, 11.51, 9.82 [02:27:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.57, 3.21, 2.77 [02:27:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.09, 11.13, 9.42 [02:28:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.85, 11.66, 10.12 [02:29:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.67, 11.01, 9.60 [02:30:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.93, 12.50, 10.64 [02:32:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.96, 11.60, 10.53 [02:36:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.89, 12.15, 11.06 [02:36:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.78, 4.36, 3.52 [02:37:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.28, 11.02, 10.05 [02:38:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.53, 11.62, 11.00 [02:38:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.33, 3.77, 3.40 [02:39:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.85, 11.24, 10.25 [02:40:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.17, 4.40, 3.68 [02:41:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.56, 10.14, 9.99 [02:42:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.31, 3.79, 3.53 [02:44:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.28, 9.20, 10.08 [02:44:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.34, 2.89, 3.23 [02:55:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.75, 4.02, 3.22 [02:57:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.37, 3.02, 2.94 [03:01:18] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 74 seconds old and 275 bytes [03:02:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.96, 3.69, 3.01 [03:03:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.37, 5.75, 4.02 [03:04:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.39, 3.69, 3.07 [03:07:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.45, 3.90, 3.65 [03:09:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.58, 3.08, 3.38 [03:16:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.78, 3.56, 3.91 [03:17:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.75, 9.47, 9.10 [03:19:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.25, 9.39, 9.13 [03:21:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.03, 10.95, 9.98 [03:22:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.44, 2.55, 3.32 [03:22:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.70, 10.31, 9.85 [03:24:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.68, 10.06, 9.84 [03:26:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.42, 10.39, 9.68 [03:28:18] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.02, 9.79, 9.54 [03:32:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.31, 11.05, 10.34 [03:34:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.95, 11.15, 10.48 [03:36:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.49, 9.68, 10.04 [03:37:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.75, 6.19, 4.46 [03:43:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.58, 3.69, 3.86 [03:49:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.85, 6.03, 4.78 [03:53:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.21, 3.48, 3.51 [03:53:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.26, 7.35, 5.62 [03:59:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.03, 3.82, 3.74 [04:01:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.19, 3.91, 3.77 [04:02:15] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:03:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.91, 3.69, 2.97 [04:04:12] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.247 second response time [04:05:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.96, 9.37, 9.07 [04:05:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.08, 3.11, 2.85 [04:05:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.27, 7.69, 7.05 [04:07:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.67, 8.15, 8.66 [04:11:22] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.03, 3.47, 3.08 [04:13:22] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.16, 3.28, 3.06 [04:13:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.01, 7.87, 7.38 [04:16:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.15, 3.30, 3.97 [04:17:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.94, 10.62, 9.54 [04:17:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.47, 7.71, 7.49 [04:18:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.24, 3.54, 3.97 [04:19:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.55, 10.43, 9.60 [04:19:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.70, 8.28, 7.73 [04:20:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.59, 3.21, 3.80 [04:23:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.56, 9.96, 9.66 [04:23:50] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.53, 7.38, 7.49 [04:25:27] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.97, 7.45, 4.52 [04:25:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.14, 7.77, 7.63 [04:27:22] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.64, 3.43, 3.09 [04:29:22] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.82, 3.48, 3.15 [04:30:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.15, 2.60, 3.28 [04:30:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.24, 10.60, 9.91 [04:31:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.17, 3.81, 3.77 [04:31:22] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.38, 3.17, 3.09 [04:34:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.82, 10.07, 9.87 [04:38:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.54, 3.73, 3.42 [04:40:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.42, 3.12, 3.22 [04:41:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.99, 2.66, 3.25 [04:41:22] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.60, 3.76, 3.28 [04:43:22] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.90, 3.77, 3.34 [04:45:22] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.96, 3.38, 3.25 [04:49:56] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.4039836824 secs [04:51:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.77, 7.13, 7.79 [04:53:56] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.003987699747 secs [04:59:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.23, 7.16, 7.51 [05:01:09] RECOVERY - db101 Backups SQL mhglobal on db101 is OK: FILE_AGE OK: /var/log/sql-mhglobal-backup-weekly.log is 67 seconds old and 0 bytes [05:01:42] RECOVERY - db112 Backups SQL reports on db112 is OK: FILE_AGE OK: /var/log/sql-reports-backup-weekly.log is 92 seconds old and 88 bytes [05:01:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.58, 7.00, 7.41 [05:02:13] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.63, 3.41, 3.16 [05:04:11] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.15, 3.33, 3.17 [05:09:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.24, 7.15, 7.24 [05:11:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.65, 7.07, 7.21 [05:19:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.10, 7.96, 7.48 [05:21:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.44, 7.60, 7.43 [05:23:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.70, 3.10, 2.67 [05:25:36] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.23, 2.72, 2.58 [05:25:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.20, 7.48, 7.34 [05:27:42] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.14, 3.74, 3.26 [05:29:40] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.90, 3.66, 3.29 [05:39:30] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.37, 3.18, 3.26 [05:43:24] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.04, 3.59, 3.40 [05:45:23] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.60, 3.20, 3.28 [05:49:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.24, 7.41, 7.92 [05:53:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.40, 7.57, 7.83 [05:55:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.89, 7.35, 7.73 [06:07:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.02, 7.15, 7.30 [06:09:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.75, 3.62, 2.79 [06:09:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.78, 7.33, 7.35 [06:11:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.04, 2.70, 2.55 [06:15:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.92, 5.70, 6.63 [06:19:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.77, 7.20, 6.98 [06:21:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.40, 6.51, 6.76 [06:25:54] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.74, 3.33, 2.93 [06:27:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.82, 7.20, 6.98 [06:28:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 3.67, 6.30, 3.49 [06:29:50] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.77, 3.25, 3.01 [06:32:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.96, 3.35, 2.91 [06:33:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.50, 6.46, 6.76 [07:16:07] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.010 second response time [07:16:15] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:17] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:36] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:19:46] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw142 [07:19:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.41, 6.12, 4.91 [07:20:06] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.526 second response time [07:20:51] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw141 mw142 [07:21:24] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:26] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.44, 5.73, 4.91 [07:21:56] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 2 backends are down. mw141 mw142 [07:22:05] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 3 backends are down. mw122 mw141 mw142 [07:22:16] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.206 second response time [07:23:19] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.217 second response time [07:23:24] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.212 second response time [07:23:24] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.280 second response time [07:23:52] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.205 second response time [07:24:51] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [07:25:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.29, 8.50, 6.65 [07:25:44] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [07:25:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.95, 6.73, 5.53 [07:25:56] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [07:26:05] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [07:27:22] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.15, 3.31, 2.56 [07:27:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.77, 7.61, 6.54 [07:29:22] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.35, 3.35, 2.67 [07:31:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.14, 6.47, 5.89 [07:37:18] !log [universalomega@puppet141] Upgraded packages libnss3 on cloud14 [07:37:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:37] !log [universalomega@puppet141] Upgraded packages libnss3 on cloud13 [07:37:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:52] !log [universalomega@puppet141] Upgraded packages libnss3 on cloud10 [07:37:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:38:05] !log [universalomega@puppet141] Upgraded packages libnss3 on cloud11 [07:38:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:38:15] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [07:38:50] !log [universalomega@puppet141] Upgraded packages libnss3 on cloud12 [07:38:50] RECOVERY - cloud13 APT on cloud13 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [07:38:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:39:12] RECOVERY - cloud14 APT on cloud14 is OK: APT OK: 94 packages available for upgrade (0 critical updates). [07:39:12] !log [universalomega@puppet141] Upgraded packages libnss3 on es131 [07:39:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:39:23] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [07:39:28] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [07:39:38] !log [universalomega@puppet141] Upgraded packages libnss3 on es141 [07:39:39] RECOVERY - es141 APT on es141 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:39:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:40:26] !log [universalomega@puppet141] Upgraded packages libnss3 on mw131 [07:40:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:40:38] RECOVERY - es131 APT on es131 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:40:50] !log [universalomega@puppet141] Upgraded packages libnss3 on mw132 [07:40:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:40:54] RECOVERY - mw131 APT on mw131 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [07:41:35] !log [universalomega@puppet141] Upgraded packages libnss3 on mwtask141 [07:41:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:41:47] !log [universalomega@puppet141] Upgraded packages libnss3 on mw141 [07:41:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:01] RECOVERY - mw132 APT on mw132 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [07:42:02] !log [universalomega@puppet141] Upgraded packages libnss3 on mw142 [07:42:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:10] RECOVERY - mw141 APT on mw141 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [07:42:15] RECOVERY - mwtask141 APT on mwtask141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:42:16] !log [universalomega@puppet141] Upgraded packages libnss3 on test131 [07:42:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:29] RECOVERY - mw142 APT on mw142 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [07:42:41] !log [universalomega@puppet141] Upgraded packages libnss3 on puppet141 [07:42:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:49] RECOVERY - test131 APT on test131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:42:55] RECOVERY - puppet141 APT on puppet141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:43:29] !log [universalomega@puppet141] Upgraded packages libnss3 on mw121 [07:43:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:43:50] !log [universalomega@puppet141] Upgraded packages libnss3 on graylog121 [07:43:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:44:09] !log [universalomega@puppet141] Upgraded packages libnss3 on mw122 [07:44:12] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [07:44:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:44:36] RECOVERY - mw121 APT on mw121 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [07:45:50] RECOVERY - mw122 APT on mw122 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [08:07:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 15.82, 7.92, 3.94 [08:13:22] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.89, 3.06, 2.98 [10:08:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.35, 4.04, 2.36 [10:10:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.46, 3.49, 2.38 [10:12:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.80, 2.85, 2.27 [10:27:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.79, 3.02, 1.79 [10:29:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.18, 3.25, 2.04 [10:38:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.41, 4.93, 2.96 [10:39:44] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:42:10] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:46:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.86, 3.27, 3.30 [11:40:39] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.3080947399 secs [11:44:40] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.01728618145 secs [12:27:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.44, 3.53, 2.05 [12:27:49] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:29:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.62, 3.42, 2.21 [12:29:50] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:31:22] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.39, 2.67, 2.08 [12:33:48] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.168684572 secs [12:39:10] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:39:48] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.04527240992 secs [12:40:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.41, 3.78, 2.26 [12:47:07] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:48:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.14, 3.33, 2.78 [12:51:49] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1694205105 secs [12:55:50] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.04316335917 secs [13:08:05] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1533455849 secs [13:08:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 18.03, 7.17, 3.73 [13:12:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.06, 3.91, 3.20 [13:14:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.90, 2.90, 2.91 [13:40:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.96, 3.68, 2.63 [13:42:06] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.05581593513 secs [13:42:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.27, 2.79, 2.43 [14:13:57] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2379438579 secs [14:25:59] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.02894279361 secs [14:57:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.59, 5.64, 3.43 [15:00:44] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7963 MB (5% inode=98%); [15:01:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.37, 3.41, 3.02 [15:03:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.57, 2.50, 2.73 [15:39:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.85, 3.85, 2.58 [15:41:51] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:43:29] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.11, 3.44, 2.76 [15:43:51] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:45:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.12, 2.70, 2.57 [16:12:14] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [16:19:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ac6e872f3841 [16:19:54] [02miraheze/puppet] 07paladox 03ac6e872 - swift: add monitoring urls to rewrite.py [16:19:57] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [16:19:58] [02puppet] 07paladox opened pull request 03#3157: swift: add monitoring urls to rewrite.py - 13https://github.com/miraheze/puppet/pull/3157 [16:24:50] [02puppet] 07paladox closed pull request 03#3157: swift: add monitoring urls to rewrite.py - 13https://github.com/miraheze/puppet/pull/3157 [16:24:53] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [16:24:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7156eb903a50...565104e227d9 [16:24:55] [02miraheze/puppet] 07paladox 03565104e - swift: add monitoring urls to rewrite.py (#3157) [16:24:56] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [16:25:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.27, 2.71, 1.91 [16:27:56] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:32:57] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:36:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.52, 3.18, 2.17 [16:37:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.40, 3.48, 3.39 [16:38:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.12, 2.76, 2.14 [16:39:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.12, 2.72, 3.12 [16:40:13] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:51:22] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.02, 2.98, 2.29 [16:55:22] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.88, 3.10, 2.52 [17:06:05] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1069611013 secs [17:21:58] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [17:40:05] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08242413402 secs [18:08:44] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8054 MB (6% inode=98%); [18:20:00] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [18:24:44] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7968 MB (5% inode=98%); [18:52:40] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1423928142 secs [18:55:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.43, 3.49, 2.59 [19:01:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.36, 3.54, 3.18 [19:03:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.25, 2.73, 2.92 [19:12:42] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09660610557 secs [19:25:52] [02ManageWiki] 07Universal-Omega reviewed pull request 03#378 commit - 13https://github.com/miraheze/ManageWiki/pull/378#discussion_r1111306258 [19:27:19] [02ManageWiki] 07Universal-Omega reviewed pull request 03#378 commit - 13https://github.com/miraheze/ManageWiki/pull/378#discussion_r1111306463 [19:27:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.98, 10.58, 9.33 [19:29:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.14, 10.97, 9.60 [19:31:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.93, 11.17, 9.78 [19:37:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.87, 10.67, 10.29 [19:37:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.21, 9.53, 9.54 [19:39:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.58, 8.90, 9.69 [20:06:39] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:08:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.84, 4.03, 2.81 [20:09:17] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:11:18] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:14:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.70, 3.45, 3.11 [20:16:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.81, 2.95, 2.96 [20:35:16] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:00:57] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.63, 3.08, 2.62 [21:02:57] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 2.44, 2.66, 2.51 [21:25:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.35, 3.87, 2.45 [21:27:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.29, 3.22, 2.38 [21:34:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.44, 9.42, 8.86 [21:36:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.06, 8.98, 8.77 [21:37:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.37, 3.82, 2.75 [21:39:34] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:41:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.13, 3.94, 3.09 [21:41:34] RECOVERY - cp32 APT on cp32 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:42:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.43, 3.11, 2.89 [22:06:31] !log db112: PURGE BINARY LOGS BEFORE '2023-02-19 23:00'; [22:06:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:44] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 28306 MB (21% inode=98%); [23:14:39] PROBLEM - pubwiki.lab.co.pl - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.40673565864563 seconds [23:44:19] RECOVERY - pubwiki.lab.co.pl - reverse DNS on sslhost is OK: SSL OK - pubwiki.lab.co.pl reverse DNS resolves to cp23.miraheze.org - CNAME OK [23:57:49] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 14.64, 6.50, 3.71