[00:02:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.96, 5.35, 3.77 [00:03:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.79, 3.94, 3.82 [00:05:09] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.08, 2.35, 3.20 [00:05:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.82, 4.12, 3.90 [00:11:42] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.75, 3.43, 3.64 [00:14:11] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 8.27, 3.70, 1.86 [00:15:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.95, 2.85, 3.34 [00:16:10] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.27, 3.79, 2.15 [00:18:10] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.64, 2.66, 1.94 [00:21:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.76, 4.19, 3.45 [00:24:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.83, 2.98, 3.90 [00:24:53] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.39, 3.14, 3.17 [00:26:45] PROBLEM - swiftobject112 Disk Space on swiftobject112 is WARNING: DISK WARNING - free space: / 32812 MB (6% inode=85%); [00:28:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 2.99, 3.64 [00:30:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.62, 3.81 [00:34:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 2.71, 3.40 [00:38:52] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.16, 2.29, 3.72 [00:44:44] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.34, 2.44, 3.36 [00:51:50] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.83, 9.93, 7.09 [00:53:46] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.70, 8.20, 6.78 [00:54:59] [02puppet] 07Universal-Omega closed pull request 03#2995: check_reverse_dns: allow cname flattening for namecheap dns - 13https://github.com/miraheze/puppet/pull/2995 [00:55:00] [url] Page not found · GitHub · GitHub | github.com [00:55:00] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e8fe6e16b5b9...b4d35d828993 [00:55:01] [url] Comparing e8fe6e16b5b9...b4d35d828993 · miraheze/puppet · GitHub | github.com [00:55:02] [02miraheze/puppet] 07MacFan4000 03b4d35d8 - check_reverse_dns: allow cname flattening for namecheap dns (#2995) [00:58:21] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 14.95, 6.86, 4.37 [00:58:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 21.56, 9.89, 5.40 [01:00:02] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.27, 4.97, 3.61 [01:00:09] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:05] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.236 second response time [01:02:55] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 3 backends are down. mw121 mw122 mw142 [01:03:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.05, 4.59, 3.21 [01:03:48] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.48, 3.26, 3.23 [01:03:59] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 4 backends are down. mw121 mw122 mw132 mw142 [01:04:13] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 2 backends are down. mw121 mw142 [01:04:17] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 3 backends are down. mw121 mw122 mw142 [01:05:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.04, 3.96, 3.16 [01:05:53] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.052 second response time [01:06:10] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [01:06:15] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [01:06:55] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [01:07:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.35, 3.70, 3.99 [01:07:52] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [01:09:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.71, 3.32, 3.09 [01:09:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.12, 3.77, 3.98 [01:10:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.33, 2.98 [01:11:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 3.26, 2.99 [01:14:56] PROBLEM - swiftobject113 Disk Space on swiftobject113 is WARNING: DISK WARNING - free space: / 32251 MB (6% inode=85%); [01:15:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.56, 3.18 [01:17:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.94, 3.37, 3.16 [01:21:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.14, 3.79, 2.90 [01:22:18] RECOVERY - swiftobject111 Disk Space on swiftobject111 is OK: DISK OK - free space: / 58724 MB (11% inode=85%); [01:23:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.33, 3.55, 2.92 [01:25:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.76, 4.08, 3.17 [01:25:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.66, 3.50, 3.14 [01:25:55] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.05, 3.12, 3.83 [01:27:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.72, 3.92, 3.43 [01:29:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.72, 3.41 [01:30:30] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.96, 3.54, 2.95 [01:31:16] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.86, 2.83, 3.02 [01:31:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.04, 3.95 [01:32:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.30, 2.62, 2.68 [01:33:44] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.87, 2.45, 3.27 [01:35:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.66, 3.46 [01:37:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.47, 3.42 [01:37:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.18, 3.67 [01:37:48] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1470910311 secs [01:38:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.23, 3.56, 3.04 [01:39:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.33, 3.66 [01:40:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.00, 2.89, 2.85 [01:41:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.86, 3.60 [01:43:39] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.78, 4.03, 3.61 [01:43:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.78, 3.80 [01:45:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.27, 3.20, 3.65 [01:47:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.17, 3.72, 3.80 [01:49:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.85, 3.84, 3.83 [01:53:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.42, 4.10, 3.93 [01:53:37] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09732747078 secs [01:55:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.09, 3.82, 3.85 [01:58:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.87, 3.29, 3.90 [01:59:13] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:20] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 0.67, 2.28, 3.23 [01:59:30] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:35] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:36] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:42] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:50] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.54, 3.67, 3.09 [02:01:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 217.174.247.33/cpweb, 109.228.51.216/cpweb, 2a00:da00:1800:326::1/cpweb, 2a00:da00:1800:328::1/cpweb, 108.175.15.182/cpweb, 74.208.203.152/cpweb, 2607:f1c0:1800:8100::1/cpweb, 2607:f1c0:1800:26f::1/cpweb [02:02:00] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 97% [02:02:11] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 2 backends are down. mw122 mw141 [02:02:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.54, 3.20, 2.98 [02:03:13] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:13] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [02:03:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 108.175.15.182/cpweb, 74.208.203.152/cpweb, 2607:f1c0:1800:8100::1/cpweb [02:03:22] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:36] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.287 second response time [02:03:40] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.514 second response time [02:03:49] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.012 second response time [02:04:01] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.446 second response time [02:04:05] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [02:04:55] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [02:04:56] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 5 backends are down. mw121 mw131 mw132 mw141 mw142 [02:05:07] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [02:05:13] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 50% [02:05:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:05:19] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3421 bytes in 0.427 second response time [02:05:22] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.297 second response time [02:05:35] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.406 second response time [02:05:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:06:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.77, 2.36, 3.24 [02:07:13] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [02:07:48] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5936 bytes in 0.015 second response time [02:07:54] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 24% [02:08:10] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [02:08:55] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [02:09:14] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 6% [02:09:44] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.418 second response time [02:10:03] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [02:10:43] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [02:15:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.48, 3.44 [02:17:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.55, 3.25, 3.36 [02:18:36] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.17, 3.56, 2.94 [02:20:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.25, 3.17, 2.85 [02:23:23] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.3041343987 secs [02:25:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.58, 3.40 [02:25:23] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.003145873547 secs [02:27:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.42, 3.36 [02:29:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 3.38, 3.35 [02:39:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.10, 3.97 [02:40:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 9.87, 8.79 [02:40:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.54, 4.17, 3.64 [02:42:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.12, 9.37, 8.74 [02:43:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.53, 4.04, 4.05 [02:44:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.93, 3.66 [02:47:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.02, 3.54, 3.93 [02:48:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.31, 4.05, 3.72 [02:52:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.54, 3.57 [02:53:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.99, 3.35, 3.95 [02:54:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.60, 3.58 [02:55:39] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.24, 3.70, 4.00 [02:55:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.44, 3.65 [02:56:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.39, 3.51 [02:58:02] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.69, 3.42, 3.00 [02:58:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.17, 4.02, 3.72 [02:59:59] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.73, 3.21, 2.98 [03:00:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.72, 3.64 [03:02:49] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2218688428 secs [03:04:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 26.91, 21.38, 10.63 [03:05:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.47, 3.07, 2.53 [03:06:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.07, 3.11, 3.39 [03:09:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.03, 2.99, 2.66 [03:11:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.42, 3.94 [03:13:23] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset -0.1151087284 secs [03:13:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.13, 4.01, 4.08 [03:15:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.10, 3.95, 3.60 [03:19:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.05, 4.34, 3.08 [03:21:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.62, 3.62, 2.96 [03:21:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.98, 4.12, 3.38 [03:23:20] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.23, 3.28, 2.92 [03:25:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.22, 2.87, 3.07 [03:27:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.58, 3.45, 3.12 [03:28:49] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.004508703947 secs [03:29:20] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.32, 3.07, 3.02 [03:31:44] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.79, 2.39, 3.82 [03:33:42] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.39, 4.07, 4.30 [03:35:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.66, 3.14, 3.93 [03:36:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.86, 4.02, 3.54 [03:37:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.89, 3.52, 3.21 [03:37:23] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.002395302057 secs [03:39:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.01, 3.58, 3.26 [03:39:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.50, 2.95, 3.94 [03:39:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 2.93, 3.90 [03:40:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.98, 2.99, 3.28 [03:41:20] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.73, 3.34, 3.22 [03:41:40] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.01, 3.82, 4.11 [03:41:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.34, 3.93 [03:43:29] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.05, 2.59, 3.40 [03:43:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.93, 3.53, 3.99 [03:43:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.33, 2.80, 3.65 [03:45:40] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.13, 4.42, 4.27 [03:45:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.22, 4.32, 4.11 [03:47:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.60, 3.64, 4.00 [03:49:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.92, 3.28, 1.96 [03:49:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.67, 3.90 [03:50:07] RECOVERY - swiftobject112 Disk Space on swiftobject112 is OK: DISK OK - free space: / 58619 MB (11% inode=86%); [03:51:39] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 3.23, 3.91, 4.03 [03:51:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.45, 4.62, 4.24 [03:52:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.25, 3.93 [03:52:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.52, 8.72, 7.72 [03:53:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.89, 3.44, 3.83 [03:54:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.92, 3.35, 3.28 [03:54:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.19, 7.67, 7.45 [03:56:09] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.28, 2.99, 3.16 [03:56:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.93, 3.57, 3.86 [03:57:19] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.16, 3.65, 2.81 [03:57:52] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1019401848 secs [03:59:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.97, 2.85, 2.63 [03:59:39] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 17.93, 7.15, 4.88 [04:01:52] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08625274897 secs [04:07:58] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.69, 3.41, 3.06 [04:09:54] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.82, 3.53, 3.15 [04:11:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.52, 3.98 [04:11:51] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 11.90, 6.19, 4.13 [04:13:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.42, 4.36, 4.23 [04:15:45] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.99, 2.63, 1.64 [04:17:42] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.68, 2.35, 1.67 [04:19:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.10, 3.01, 3.81 [04:19:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 3.28, 3.86 [04:20:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.40, 3.94 [04:22:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.05, 4.28, 4.18 [04:25:23] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.77, 3.68, 3.92 [04:25:39] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.08, 2.47, 3.30 [04:26:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.31, 3.65, 2.98 [04:26:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.60, 3.91 [04:27:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.87, 3.90, 3.95 [04:27:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.64, 2.46, 3.25 [04:28:22] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.69, 3.16, 2.88 [04:28:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.32, 4.03, 4.02 [04:29:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.27, 3.63, 3.84 [04:30:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.37, 3.77 [04:31:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.61, 3.73, 3.83 [04:31:39] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.10, 5.70, 4.38 [04:31:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.30, 6.46, 4.84 [04:32:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.01, 3.46, 3.10 [04:33:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.60, 3.44, 3.72 [04:34:12] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.37, 2.95, 2.97 [04:35:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.95, 3.78, 3.91 [04:37:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.28, 3.85 [04:39:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.37, 3.76, 3.74 [04:40:49] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2573058307 secs [04:40:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.47, 3.95, 3.72 [04:43:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.55, 3.57, 3.72 [04:43:39] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.35, 2.73, 3.38 [04:46:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.97, 3.86 [04:47:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.09, 3.31, 3.54 [04:47:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.41, 2.79, 3.37 [04:48:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.90, 3.84 [04:49:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.83, 3.19, 3.47 [04:50:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.70, 3.77 [04:51:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.91, 2.85, 2.22 [04:51:20] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.54, 2.66, 3.25 [04:53:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.61, 3.20, 2.46 [04:54:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.29, 3.78, 3.74 [04:58:49] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.0175947547 secs [04:58:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.80, 3.73 [05:02:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.93, 3.38, 3.42 [05:02:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.02, 3.40 [05:04:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.84, 5.39, 3.70 [05:04:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.43, 2.90, 3.25 [05:06:22] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.20, 3.43, 3.28 [05:08:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.56, 3.84, 3.44 [05:08:20] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.12, 2.56, 2.97 [05:10:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.90, 2.78, 3.10 [05:18:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.97, 3.57 [05:22:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 3.74, 3.52 [05:24:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.91, 3.01, 3.28 [05:25:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.02, 3.01, 2.32 [05:25:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.34, 3.11 [05:27:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.39, 2.41, 2.19 [05:27:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.31, 3.74, 3.26 [05:29:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.62, 3.28 [05:31:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.16, 3.40, 3.24 [05:33:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.91, 4.34, 2.46 [05:35:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.05, 6.76, 4.32 [05:35:19] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.05, 3.98, 2.57 [05:37:10] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.95, 3.63, 2.74 [05:37:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.60, 2.73, 2.28 [05:39:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.25, 3.36, 2.75 [05:43:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.68, 3.32, 3.82 [05:47:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 15.68, 6.29, 4.64 [05:49:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.46, 3.17 [05:51:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 3.17, 3.10 [05:55:04] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.82, 3.53, 3.90 [05:55:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.45, 4.07, 3.46 [05:55:38] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 8.21, 12.24, 8.12 [05:56:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.71, 3.23, 2.85 [05:56:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.57, 3.12 [05:57:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 7.48, 4.93, 3.25 [05:57:38] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.86, 9.99, 7.79 [05:58:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.23, 4.17, 3.38 [05:59:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.37, 2.36, 3.32 [05:59:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.77, 3.23, 2.96 [06:00:49] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 2.69, 4.69, 3.75 [06:00:56] RECOVERY - swiftobject113 Disk Space on swiftobject113 is OK: DISK OK - free space: / 58906 MB (11% inode=86%); [06:01:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.19, 3.57, 3.50 [06:01:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.63, 4.00, 3.33 [06:02:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.86, 3.48 [06:02:46] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.94, 3.87, 3.56 [06:03:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 3.14, 3.35 [06:03:20] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 0.98, 2.97, 3.04 [06:04:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.56, 3.05, 3.23 [06:04:44] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.27, 2.92, 3.25 [06:23:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.02, 2.95 [06:25:17] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.41, 5.51, 3.62 [06:25:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.96, 3.51, 3.13 [06:27:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 3.27, 3.09 [06:28:09] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 8.14, 4.98, 2.49 [06:29:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.12, 3.67, 2.07 [06:30:06] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.54, 3.89, 2.38 [06:31:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.41, 3.32, 3.26 [06:31:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.93, 2.56, 1.85 [06:31:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.40, 3.75, 4.00 [06:32:04] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.29, 2.87, 2.18 [06:35:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.58, 4.08, 4.06 [06:39:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.31, 3.81, 3.96 [06:46:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.14, 2.86 [06:47:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.39, 4.00, 3.99 [06:48:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.81, 3.83, 3.12 [06:50:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.49, 3.08 [06:54:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.05, 3.08, 3.03 [07:00:16] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.73, 4.17, 3.06 [07:02:13] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.15, 3.41, 2.92 [07:04:10] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.84, 3.23, 2.91 [07:05:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 2.89, 4.38, 3.04 [07:07:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.07, 3.22, 2.78 [07:09:59] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.51, 3.14, 2.90 [07:13:36] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.124504447 secs [07:13:54] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.89, 4.69, 3.47 [07:15:51] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.16, 3.92, 3.33 [07:17:48] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.96, 3.28, 3.17 [07:22:51] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.83, 2.83, 1.98 [07:24:51] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.78, 2.43, 1.95 [07:29:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.69, 3.71, 3.99 [07:31:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.29, 4.02, 4.07 [07:37:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.59, 3.26, 2.17 [07:39:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.81, 3.35, 2.37 [07:42:59] PROBLEM - cp23 APT on cp23 is UNKNOWN: NRPE: Command timed out after 60 seconds [07:43:36] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.007235825062 secs [07:45:04] PROBLEM - cp23 APT on cp23 is CRITICAL: APT CRITICAL: 17 packages available for upgrade (8 critical updates). [07:55:11] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.69, 3.25, 2.83 [07:57:08] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.10, 3.21, 2.87 [08:06:03] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.23, 3.76, 2.59 [08:07:57] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.51, 2.95, 2.43 [08:15:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.24, 3.61, 3.88 [08:20:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.90, 3.23 [08:22:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.72, 3.23 [08:26:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.99, 3.30, 3.17 [08:27:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.45, 3.87, 3.86 [08:29:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 4.00, 3.90, 3.87 [08:31:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.00, 4.15, 3.98 [08:34:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.29, 3.62, 4.00 [08:36:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.40, 4.77, 3.16 [08:36:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.07, 3.70, 3.98 [08:38:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.49, 3.74, 2.99 [08:38:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.29, 3.55, 3.89 [08:39:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.92, 3.74, 3.94 [08:40:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.31, 3.68, 3.89 [08:42:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.52, 2.91, 2.86 [08:43:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.37, 3.95, 3.96 [08:43:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.23, 3.12 [08:45:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.89, 2.69, 2.94 [08:53:26] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:53:44] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.32, 3.31, 2.38 [08:55:41] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.71, 3.51, 2.56 [08:57:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.62, 3.24, 2.58 [09:04:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.17, 3.54, 3.92 [09:08:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.33, 3.63, 2.51 [09:08:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.99, 4.25, 4.12 [09:09:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.11, 3.69, 3.96 [09:14:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.89, 3.89, 3.99 [09:15:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.43, 3.93, 3.95 [09:16:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.76, 3.71, 3.18 [09:16:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.83, 5.03, 4.43 [09:18:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.17, 2.81, 2.91 [09:19:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.41, 3.77, 3.90 [09:21:26] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:25:58] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.10, 3.36, 2.44 [09:26:49] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1883105934 secs [09:29:52] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.59, 2.86, 2.48 [09:33:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.23, 4.05, 3.89 [09:34:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.89, 4.06, 2.64 [09:34:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.58, 4.15, 2.75 [09:37:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.36, 3.85, 3.86 [09:37:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.07, 3.88, 3.02 [09:39:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.55, 4.04, 3.92 [09:39:54] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.50, 4.00, 3.31 [09:39:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.44, 3.05, 2.82 [09:41:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.59, 3.82, 3.86 [09:41:51] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.47, 3.15, 3.08 [09:50:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.55, 3.75, 3.99 [09:53:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.34, 3.90, 3.80 [09:54:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.16, 3.73, 3.91 [09:54:49] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.05172273517 secs [09:54:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.69, 3.83, 3.99 [09:55:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.66, 3.79, 3.77 [09:58:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.70, 3.72, 3.86 [09:58:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.23, 3.82, 3.94 [10:00:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.34, 3.86, 3.89 [10:00:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.82, 3.79, 3.91 [10:02:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.11, 3.57, 3.78 [10:03:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.65, 4.06, 3.84 [10:07:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.34, 3.87, 3.84 [10:14:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.07, 3.97, 3.83 [10:14:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.06, 3.85, 3.81 [10:15:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.46, 3.75, 3.73 [10:18:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.50, 3.95, 3.88 [10:18:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.00, 3.71, 3.79 [10:19:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.51, 11.63, 7.67 [10:19:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.37, 3.72, 3.74 [10:21:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.40, 9.77, 7.49 [10:22:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.86, 4.02, 3.89 [10:22:49] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset -0.1374179423 secs [10:24:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.86, 3.75, 2.26 [10:24:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.67, 3.84, 3.84 [10:25:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.03, 3.57, 3.63 [10:26:31] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.97, 3.43, 2.33 [10:26:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.01, 3.97, 3.89 [10:26:49] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.08098331094 secs [10:27:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.64, 3.58, 3.63 [10:28:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.89, 4.41, 2.80 [10:28:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.48, 3.75, 3.82 [10:28:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.69, 3.22 [10:28:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.49, 3.82, 3.76 [10:29:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.22, 3.88, 3.73 [10:30:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.33, 3.15 [10:31:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.82, 3.83, 3.73 [10:32:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.38, 3.17, 2.70 [10:33:01] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.130682826 secs [10:34:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.13, 3.88, 3.82 [10:35:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 6.05, 4.59, 4.01 [10:37:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.52, 3.15, 2.13 [10:38:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.76, 3.99, 3.95 [10:39:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.39, 3.10, 2.26 [10:42:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.03, 4.06, 3.95 [10:43:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.96, 3.83, 3.92 [10:44:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.45, 3.24 [10:48:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.67, 3.88, 3.99 [10:48:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.88, 3.11, 3.16 [10:48:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.97, 3.97, 3.96 [10:50:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.18, 4.00, 4.02 [10:52:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.36, 3.93, 3.93 [10:52:58] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09340062737 secs [10:53:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.24, 4.18, 3.94 [10:55:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.43, 3.87, 3.86 [10:55:39] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.43, 4.03, 2.91 [10:55:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 4.20, 3.06 [10:57:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.48, 3.53, 2.87 [10:57:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.72, 3.49, 2.96 [10:58:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.51, 3.91, 3.94 [10:59:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.14, 4.73, 3.44 [11:00:15] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 14.23, 6.66, 4.06 [11:01:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.65, 3.56, 2.53 [11:02:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.65, 4.12, 4.00 [11:03:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.82, 3.21, 2.53 [11:03:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.17, 3.72, 3.79 [11:03:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.54, 3.88, 3.51 [11:04:10] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.23, 3.76, 3.47 [11:05:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 0.89, 2.85, 3.18 [11:06:07] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 0.80, 2.74, 3.12 [11:06:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.14, 3.78, 3.90 [11:07:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.43, 3.90, 3.89 [11:10:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.22, 3.80, 3.86 [11:12:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.12, 3.48, 3.73 [11:13:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.78, 3.74, 3.75 [11:15:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.18, 3.45, 3.64 [11:20:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.71, 3.83, 4.00 [11:20:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.73, 4.08, 3.91 [11:22:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.99, 3.70, 3.80 [11:27:38] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.69, 3.07, 3.35 [11:27:39] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 13.92, 6.23, 3.67 [11:27:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 24.20, 9.70, 4.92 [11:29:14] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:34] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:37] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:42] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:13] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:49] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:56] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 5 backends are down. mw121 mw122 mw132 mw141 mw142 [11:31:10] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.349 second response time [11:31:31] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.281 second response time [11:31:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.284 second response time [11:31:50] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [11:32:03] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [11:32:10] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.455 second response time [11:32:29] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [11:32:46] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.223 second response time [11:33:13] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 42% [11:33:44] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.376 second response time [11:34:10] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [11:34:27] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 6% [11:35:13] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 7% [11:36:55] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [11:37:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.45, 4.00, 3.60 [11:37:50] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [11:38:03] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [11:38:10] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [11:38:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.24, 3.46, 3.52 [11:39:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.04, 3.65, 3.52 [11:40:46] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.07, 3.26, 3.44 [11:41:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.19, 4.09, 3.69 [11:42:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.28, 3.78, 3.68 [11:42:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.559 second response time [11:42:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.42, 3.61, 3.54 [11:43:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.96, 3.95, 3.69 [11:43:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 3.46, 2.92 [11:44:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.35, 3.48, 3.50 [11:45:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.71, 3.09 [11:46:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.46, 3.83, 3.74 [11:48:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.13, 3.96, 3.80 [11:48:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.33, 3.82, 3.63 [11:49:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 3.06, 2.99 [11:49:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.50, 4.20, 3.84 [11:51:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.32, 3.94, 3.80 [11:52:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.44, 3.86, 3.71 [11:54:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.34, 3.80, 3.81 [11:56:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.16, 4.25, 3.97 [11:57:39] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.38, 2.06, 3.93 [11:59:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.46, 3.10 [11:59:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.47, 3.74, 2.35 [12:00:50] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.25, 3.44, 2.53 [12:01:18] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.46, 2.84, 2.19 [12:01:40] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.44, 1.68, 3.35 [12:02:44] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.21, 2.72, 2.38 [12:03:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.42, 1.85, 3.81 [12:05:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.82, 3.35 [12:05:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.04, 4.17, 3.85 [12:06:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.52, 3.92, 3.74 [12:06:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.46, 3.27 [12:07:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.45, 3.85, 3.77 [12:07:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.42, 1.77, 3.34 [12:08:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.86, 3.45 [12:08:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.74, 3.48, 3.60 [12:10:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 3.32, 3.30 [12:16:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.28, 3.97, 3.75 [12:19:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.85, 4.42, 3.98 [12:22:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.17, 3.91, 3.85 [12:28:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.66, 4.26, 3.96 [12:29:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.72, 3.96, 3.94 [12:31:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.03, 4.17, 4.03 [12:31:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.06, 2.79 [12:32:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.15, 3.87, 3.88 [12:33:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.94, 2.65, 2.68 [12:34:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.03, 4.07, 3.96 [12:36:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.28, 3.96, 3.95 [12:38:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.72, 4.05, 3.97 [12:39:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.47, 3.29 [12:39:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.41, 3.87, 3.99 [12:40:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.50, 3.96, 2.83 [12:41:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 3.11, 3.18 [12:42:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.60, 3.77, 2.90 [12:42:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.63, 3.87, 3.91 [12:43:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.42, 3.81, 3.91 [12:44:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.08, 3.11, 2.77 [12:45:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.53, 3.62, 3.82 [12:46:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.71, 4.27, 4.06 [12:50:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.69, 3.42 [12:51:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.87, 3.95, 3.86 [12:53:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.84, 3.76, 3.79 [12:54:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.71, 3.48 [12:55:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.07, 3.04, 2.30 [12:56:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.77, 3.52 [12:57:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.43, 4.42, 4.02 [12:59:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.49, 3.14, 2.57 [13:01:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.12, 3.91, 3.91 [13:02:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.91, 3.60, 3.55 [13:03:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.39, 4.05, 3.96 [13:04:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 3.14, 3.38 [13:05:05] PROBLEM - wiki.wisrail.de - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wisrail.de All nameservers failed to answer the query. [13:08:05] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/3b25afd23868 [13:08:09] [02miraheze/puppet] 07Reception123 033b25afd - allow direct pushing of certs to GitHub T7582 [13:08:10] [02puppet] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/puppet [13:08:10] [url] Page not found · GitHub · GitHub | github.com [13:08:12] [02puppet] 07Reception123 opened pull request 03#2996: allow direct pushing of certs to GitHub T7582 - 13https://github.com/miraheze/puppet/pull/2996 [13:08:12] [url] Page not found · GitHub · GitHub | github.com [13:08:28] [02puppet] 07Reception123 synchronize pull request 03#2996: allow direct pushing of certs to GitHub T7582 - 13https://github.com/miraheze/puppet/pull/2996 [13:08:29] [url] Page not found · GitHub · GitHub | github.com [13:08:31] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3b25afd23868...55e5435fdcc9 [13:08:32] [url] Comparing 3b25afd23868...55e5435fdcc9 · miraheze/puppet · GitHub | github.com [13:08:32] [02miraheze/puppet] 07Reception123 0355e5435 - Update ssl-certificate.py [13:13:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.36, 3.78, 3.95 [13:18:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 1.57, 2.95, 3.78 [13:21:30] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.47, 2.87, 2.13 [13:23:24] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.49, 2.85, 2.22 [13:24:56] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 1.88, 2.28, 3.22 [13:27:20] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.99, 3.51, 2.62 [13:27:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.37, 3.86, 3.84 [13:29:20] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.97, 2.86, 2.49 [13:29:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.15, 3.61, 3.75 [13:31:39] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.28, 3.21, 2.30 [13:32:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.74, 3.03 [13:33:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.27, 4.02, 3.86 [13:33:39] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.69, 2.90, 2.29 [13:33:41] RECOVERY - wiki.wisrail.de - reverse DNS on sslhost is OK: SSL OK - wiki.wisrail.de reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:34:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 3.36, 2.99 [13:35:19] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.97, 2.82, 2.33 [13:37:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.27, 2.23, 2.17 [13:39:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.77, 3.54, 2.47 [13:41:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.69, 3.92, 3.99 [13:41:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.83, 3.81, 2.71 [13:42:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.53, 9.87, 8.50 [13:43:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.85, 3.00, 2.54 [13:44:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.14, 9.10, 8.41 [13:47:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.30, 3.55, 3.31 [13:48:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.88, 3.29, 2.87 [13:48:59] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 3.13, 3.19 [13:49:38] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.58, 2.81, 3.38 [13:51:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.15, 4.44, 2.99 [13:51:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 033 commits to 03master [+4/-0/±1] 13https://github.com/miraheze/ssl/compare/6cafe4609c0b...1153fbb6037b [13:51:25] [url] Comparing 6cafe4609c0b...1153fbb6037b · miraheze/ssl · GitHub | github.com [13:51:27] [02miraheze/ssl] 07MirahezeSSLBot 03998973f - Bot: Add SSL cert for .betaheze.org [13:51:28] [02miraheze/ssl] 07MirahezeSSLBot 034d83b49 - Merge branch 'master' of github.com:miraheze/ssl [13:51:31] [02miraheze/ssl] 07MirahezeSSLBot 031153fbb - Bot: Add SSL cert for .com [13:51:46] well.. at least it half worked [13:52:54] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1153fbb6037b...34dcc2752af6 [13:52:55] [url] Comparing 1153fbb6037b...34dcc2752af6 · miraheze/ssl · GitHub | github.com [13:52:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.11, 3.88, 3.19 [13:52:57] [02miraheze/ssl] 07Reception123 0334dcc27 - add tinkrpedia.com cert to certs.yaml [13:53:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.42, 3.27, 2.74 [13:53:57] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/34dcc2752af6...66d92dba7d3d [13:53:58] [url] Comparing 34dcc2752af6...66d92dba7d3d · miraheze/ssl · GitHub | github.com [13:53:59] [02miraheze/ssl] 07Reception123 0366d92db - rm test.betaheze.org (added during testing) [13:54:26] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.17, 3.26, 2.57 [13:56:28] [02puppet] 07Reception123 synchronize pull request 03#2996: allow direct pushing of certs to GitHub T7582 - 13https://github.com/miraheze/puppet/pull/2996 [13:56:28] [url] Page not found · GitHub · GitHub | github.com [13:56:29] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/55e5435fdcc9...17e76a8f3a4b [13:56:29] [url] Comparing 55e5435fdcc9...17e76a8f3a4b · miraheze/puppet · GitHub | github.com [13:56:31] [02miraheze/puppet] 07Reception123 0317e76a8 - make changes after first test [13:56:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.23, 3.86, 3.37 [13:57:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 6.38, 4.03, 3.59 [13:57:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.61, 3.19 [13:58:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.89, 3.04, 2.69 [13:58:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.40, 4.17, 3.55 [13:59:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.79, 3.56, 3.48 [13:59:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.84, 2.96, 3.01 [14:01:38] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.77, 3.29, 3.38 [14:02:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.92, 3.83, 3.03 [14:02:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.95, 3.50, 3.19 [14:03:17] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:04:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.44, 3.99, 3.69 [14:05:57] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.92, 3.14, 2.94 [14:06:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 3.20, 3.17 [14:06:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.19, 2.76, 3.87 [14:06:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.17, 4.49, 3.91 [14:08:33] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:08:41] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:08:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.08, 3.97, 3.79 [14:09:02] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:10:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 6.70, 3.94, 4.02 [14:10:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.04, 4.39, 3.96 [14:12:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.71, 10.38, 9.37 [14:12:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.82, 3.77, 2.94 [14:13:11] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:13:20] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:13:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.75, 3.60, 3.34 [14:14:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.25, 9.37, 9.16 [14:14:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.15, 2.75, 2.66 [14:14:56] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:15:06] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:17] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:15:40] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.61, 5.44, 3.86 [14:16:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 5.38, 4.20 [14:16:21] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:16:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.39, 3.73, 3.90 [14:17:03] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.376 second response time [14:17:15] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:17:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.12, 3.75, 3.45 [14:18:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.72, 3.25 [14:18:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.07, 3.83, 3.91 [14:20:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.18, 3.64, 3.84 [14:22:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 4.05, 3.46 [14:24:41] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/66d92dba7d3d...f9ea6cd72430 [14:24:42] [url] Comparing 66d92dba7d3d...f9ea6cd72430 · miraheze/ssl · GitHub | github.com [14:24:43] [02miraheze/ssl] 07MacFan4000 03f9ea6cd - Fix [14:24:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.45, 3.33 [14:26:25] PROBLEM - tinkrpedia.com.crt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tinkrpedia.com.crt could not be found [14:26:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.99, 3.36, 3.31 [14:27:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 5.29, 4.17, 3.94 [14:27:21] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.42, 3.99, 3.86 [14:29:19] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.67, 5.65, 4.42 [14:29:43] ^ ah yes CosmicAlpha was saying that my change to the monitoring needs to be manually applied [14:30:31] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.47, 3.84, 3.85 [14:30:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.47, 3.41 [14:30:52] PROBLEM - cp23 Puppet on cp23 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:31:14] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:31:46] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/tinkrpedia.com.crt.crt],File[/etc/ssl/private/tinkrpedia.com.crt.key] [14:32:31] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.04, 3.90, 3.87 [14:32:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.68, 9.81, 9.10 [14:32:49] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:35:13] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.29, 3.39, 3.78 [14:35:27] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [14:35:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.55, 3.38, 4.00 [14:36:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.39, 3.07, 3.29 [14:36:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.18, 9.77, 9.25 [14:36:45] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:37:11] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.49, 3.73, 3.85 [14:38:41] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:39:09] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.25, 3.16, 3.63 [14:39:20] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [14:40:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.17, 10.21, 9.54 [14:42:00] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:42:26] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:42:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.51, 11.06, 9.93 [14:43:07] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [14:43:25] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:44:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.13, 3.41, 3.45 [14:44:40] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [14:44:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.56, 10.34, 9.81 [14:44:55] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.26, 5.04, 4.17 [14:46:56] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.267 second response time [14:47:24] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.15, 3.91, 3.81 [14:48:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.01, 3.19, 3.35 [14:49:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.55, 3.72, 3.75 [14:50:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.35, 9.61, 9.82 [14:52:55] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.02, 3.64, 3.83 [14:57:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.34, 3.32 [14:57:54] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:58:05] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.42, 4.07, 3.97 [14:58:25] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:59:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.66, 2.82, 3.14 [14:59:25] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:59:58] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 1.81, 2.95, 3.75 [15:00:06] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.75, 3.02, 3.39 [15:03:49] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.05, 3.31, 3.69 [15:04:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.09, 3.64, 3.99 [15:04:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 10.16, 9.51 [15:06:30] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.401 second response time [15:06:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.33, 10.58, 9.85 [15:07:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.79, 9.20, 9.24 [15:08:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.13, 3.39, 3.80 [15:10:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.26, 3.03, 3.62 [15:10:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.18, 3.82 [15:12:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.11, 9.12, 9.57 [15:14:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.12, 3.66, 3.71 [15:14:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 4.06, 4.03 [15:17:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.58, 3.57, 3.37 [15:18:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.78, 3.73, 3.77 [15:18:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.87, 3.96 [15:19:16] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.70, 3.49, 3.36 [15:21:10] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.49, 3.15, 3.25 [15:21:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.42, 3.21 [15:23:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.95, 3.27, 3.17 [15:24:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.45, 4.33, 4.01 [15:24:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.28, 4.20, 4.03 [15:26:42] RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Wed 18 Jan 2023 08:23:18 GMT +0000. [15:30:31] RECOVERY - programming.red - reverse DNS on sslhost is OK: SSL OK - programming.red reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [15:30:48] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.86, 3.28, 3.89 [15:32:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.79, 3.78, 3.99 [15:32:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.42, 8.63, 8.43 [15:34:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.92, 8.31, 8.33 [15:36:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.21, 3.51, 3.34 [15:37:16] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Wed 18 Jan 2023 08:23:18 GMT +0000. [15:38:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.40, 3.04, 3.18 [15:40:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.04, 10.66, 9.29 [15:44:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.28, 3.41, 3.95 [15:44:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.79, 10.79, 9.78 [15:45:26] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.50, 3.37, 3.27 [15:45:56] RECOVERY - www.programming.red - reverse DNS on sslhost is OK: SSL OK - www.programming.red reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [15:46:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.36, 3.15, 3.84 [15:48:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.14, 3.86, 4.01 [15:50:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.21, 9.26, 9.47 [15:51:10] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.45, 3.15, 3.22 [15:52:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.22, 10.51, 9.47 [15:54:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.72, 3.67, 3.94 [15:54:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.10, 10.91, 9.76 [15:54:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.19, 3.66, 3.72 [15:55:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.16, 3.74, 2.64 [15:56:52] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.90, 3.46, 3.65 [15:58:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.43, 3.81, 3.92 [15:58:50] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.85, 3.70, 3.69 [16:00:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.44, 3.46, 3.60 [16:00:59] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:08] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.02, 3.31, 2.91 [16:02:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.74, 4.08, 3.79 [16:03:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.911 second response time [16:04:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.57, 12.37, 10.85 [16:04:46] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.48, 3.80, 3.72 [16:05:28] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2103733122 secs [16:07:14] PROBLEM - www.dlfm-wiki.top - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'dlfm-wiki.top' expires in 7 day(s) (Sat 19 Nov 2022 16:02:27 GMT +0000). [16:07:25] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.17, 4.13, 3.55 [16:08:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.17, 3.45, 3.83 [16:08:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 11.98, 11.15 [16:08:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.31, 4.13, 3.84 [16:08:50] PROBLEM - gluster122 Disk Space on gluster122 is WARNING: DISK WARNING - free space: / 57736 MB (10% inode=72%); [16:09:04] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.87, 4.07, 3.07 [16:09:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.94, 3.64, 3.43 [16:10:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.19, 4.82, 4.31 [16:11:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.58, 3.18 [16:12:13] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.41, 3.89 [16:12:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.87, 10.60, 10.01 [16:12:40] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.10, 3.83, 3.80 [16:12:59] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.76, 3.79, 3.20 [16:14:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.24, 9.16, 10.12 [16:14:19] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.325 second response time [16:14:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.11, 4.09 [16:14:35] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.86, 3.11, 2.16 [16:14:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.29, 4.32, 3.98 [16:14:56] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.70, 3.07, 3.01 [16:15:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.54, 3.76, 3.52 [16:16:34] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.63, 2.54, 2.07 [16:17:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 3.19, 3.17 [16:18:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.25, 3.48, 3.89 [16:20:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.32, 4.17, 4.09 [16:20:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.78, 9.98, 10.03 [16:22:32] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.59, 3.99, 3.98 [16:22:42] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.54, 3.88, 3.73 [16:24:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.36, 3.23 [16:26:40] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.83, 3.37, 3.16 [16:27:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.64, 11.53, 10.53 [16:28:28] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.24, 3.60, 3.75 [16:28:40] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 3.25, 3.33, 3.17 [16:28:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.96, 3.49 [16:29:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 10.37, 10.21 [16:30:27] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.82, 3.63, 3.74 [16:30:39] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.48, 3.37 [16:31:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.45, 9.65, 9.98 [16:32:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.46, 10.80, 10.21 [16:32:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.02, 3.13, 3.40 [16:32:43] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.455 second response time [16:32:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 3.09, 3.24 [16:34:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.11, 10.90, 10.32 [16:35:28] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.08158898354 secs [16:36:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.87, 11.58, 10.64 [16:37:18] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.52, 2.94, 2.35 [16:38:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.21, 11.45, 10.70 [16:38:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.00, 3.35, 3.45 [16:39:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.12, 2.26, 2.17 [16:40:20] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.04, 3.82, 3.77 [16:40:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.75, 4.54, 3.25 [16:40:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.91, 3.86, 3.62 [16:40:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 10.90, 10.38 [16:42:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.65, 3.92, 3.16 [16:42:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.63, 3.63, 3.55 [16:42:57] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.50, 2.98, 2.90 [16:44:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.40, 10.60, 10.39 [16:44:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.87, 3.62, 3.97 [16:45:17] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.27, 3.46, 2.46 [16:46:16] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.23, 3.41, 3.68 [16:47:06] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.401 second response time [16:47:16] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.86, 3.17, 2.47 [16:50:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.03, 3.44, 3.61 [16:50:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.75, 11.36, 10.81 [16:50:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.79, 3.73, 3.55 [16:51:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.14, 3.97, 2.93 [16:52:11] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.55, 3.52, 3.62 [16:53:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [16:53:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.92, 3.85, 3.00 [16:54:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.92, 11.47, 11.01 [16:54:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 1.87, 3.32, 3.49 [16:54:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 1.63, 2.68, 3.38 [16:55:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.58, 2.92, 2.76 [16:55:35] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.29, 11.06, 10.83 [16:56:08] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 1.41, 2.47, 3.17 [16:56:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.77, 12.07, 11.29 [16:58:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.89, 11.52, 11.04 [16:58:34] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw121 [16:58:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.21, 4.24, 3.83 [16:59:36] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.37, 10.88, 11.10 [17:00:35] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [17:00:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.53, 3.61, 3.65 [17:01:48] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.649 second response time [17:02:40] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.55, 3.86, 3.35 [17:04:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.45, 4.20, 3.83 [17:04:40] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.83, 3.54, 3.30 [17:05:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.585 second response time [17:05:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.88, 4.23, 3.66 [17:06:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.16, 11.87, 11.24 [17:07:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.47, 3.54, 3.35 [17:08:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.47, 11.01, 10.98 [17:08:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.57, 4.44, 3.20 [17:08:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.51, 3.90, 3.79 [17:08:40] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.49, 3.28, 3.27 [17:09:55] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.87, 3.65, 3.56 [17:10:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.05, 3.68, 3.08 [17:11:12] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.11, 3.54, 3.41 [17:11:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.53, 11.65, 10.99 [17:11:54] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.53, 3.19, 3.40 [17:12:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.18, 10.78, 10.84 [17:12:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.38, 2.77, 2.81 [17:12:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.22, 3.96 [17:13:08] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 1.61, 2.85, 3.17 [17:13:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.95, 11.03, 10.85 [17:14:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.80, 3.12, 3.92 [17:14:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.08, 10.53, 10.74 [17:14:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.43, 3.93 [17:14:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.56, 3.84, 3.74 [17:14:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.58, 3.19 [17:15:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.43, 3.77, 3.57 [17:16:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.64, 3.57, 3.98 [17:16:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.27, 3.81 [17:16:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.33, 3.67, 3.69 [17:16:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.77, 3.30, 3.14 [17:18:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.75, 3.29, 3.83 [17:18:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.54, 3.84 [17:19:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.83, 3.71, 3.62 [17:20:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.60, 3.84 [17:22:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.54, 4.45, 4.17 [17:22:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.59, 9.38, 10.12 [17:22:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.25, 3.08, 3.39 [17:24:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.58, 4.61, 4.17 [17:25:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.53, 10.73, 10.58 [17:27:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.66, 3.04, 3.33 [17:28:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 3.95, 3.45 [17:29:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.85, 11.11, 10.81 [17:30:16] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.75, 3.50 [17:33:33] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.95, 4.27, 3.51 [17:35:30] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.06, 3.76, 3.41 [17:35:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.22, 8.53, 9.81 [17:36:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.94, 3.60, 2.67 [17:36:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.318 second response time [17:36:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.10, 3.37, 3.39 [17:38:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [17:38:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.03, 3.04, 2.58 [17:39:25] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.04, 3.70, 3.45 [17:39:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.42, 9.50, 9.92 [17:41:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.68, 11.19, 10.49 [17:42:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.39, 10.23, 10.14 [17:42:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.41, 3.33, 3.98 [17:42:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 5.67, 4.40, 3.62 [17:43:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.60, 11.10, 10.54 [17:44:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.21, 3.32, 3.88 [17:44:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.35, 9.05, 9.71 [17:44:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.35, 3.95, 4.12 [17:45:26] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 6.25, 4.62, 3.63 [17:46:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.55, 4.01, 4.08 [17:47:16] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.25, 3.86, 3.64 [17:48:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.68, 3.51, 3.89 [17:48:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.63, 4.00 [17:48:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.96, 3.82, 3.66 [17:49:13] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 3.96, 4.10, 3.77 [17:49:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.14, 9.37, 9.98 [17:50:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.02, 4.14, 3.81 [17:50:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.27, 3.55, 3.24 [17:51:10] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.79, 3.95, 3.69 [17:52:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.25, 3.83, 3.93 [17:52:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.08, 3.80, 3.72 [17:52:43] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.15, 3.36, 3.20 [17:53:09] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.25, 3.32, 3.54 [17:54:03] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.59, 3.82, 3.91 [17:54:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.79, 9.82, 9.57 [17:54:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 3.81, 3.89 [17:54:52] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1329428554 secs [17:55:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.33, 3.30, 2.56 [17:56:03] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.40, 4.77, 4.25 [17:56:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.88, 9.48, 9.48 [17:56:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.51, 3.78 [17:56:54] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.06, 2.85, 3.29 [17:57:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.85, 2.71, 2.44 [17:57:04] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.15, 2.85, 3.30 [18:00:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.74, 5.19, 4.34 [18:00:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.92, 2.94, 3.33 [18:03:02] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:17] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:06] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.564 second response time [18:08:15] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.505 second response time [18:08:28] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.30, 3.60, 3.40 [18:12:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.76, 10.49, 9.87 [18:12:52] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08461475372 secs [18:14:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.26, 10.11, 9.82 [18:16:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.78, 4.21, 3.59 [18:17:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.92, 10.30, 9.62 [18:17:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.19, 3.75, 3.44 [18:17:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.04, 10.22, 9.94 [18:18:21] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.34, 3.09, 3.30 [18:18:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [18:19:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.97, 3.39 [18:21:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.79, 9.95, 9.65 [18:21:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.75, 3.37 [18:21:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.61, 11.07, 10.31 [18:22:17] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.50, 4.04, 3.63 [18:23:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.53, 10.67, 10.27 [18:24:17] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.66, 3.81, 3.59 [18:24:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 2.96, 3.82 [18:25:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.89, 9.65, 9.94 [18:27:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.22, 3.66, 3.70 [18:28:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.52, 3.17, 3.40 [18:30:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.68, 3.81 [18:30:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.67, 3.67, 3.91 [18:30:54] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 1.42, 2.63, 3.28 [18:31:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.94, 3.20, 3.29 [18:32:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.83, 3.87 [18:33:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [18:36:25] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.55, 3.72 [18:38:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.202 second response time [18:38:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.00, 2.63, 3.31 [18:48:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.46, 9.83, 9.45 [18:48:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.75, 10.74, 10.00 [18:50:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.47, 9.77, 9.48 [18:50:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.80, 3.27, 3.85 [18:52:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.79, 4.13, 4.08 [18:53:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.85, 3.44, 2.49 [18:54:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.59, 10.12, 10.03 [18:54:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.90, 3.64, 3.90 [18:54:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.45, 3.82, 3.42 [18:55:29] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.40, 2.65, 2.32 [18:56:42] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.32, 3.19, 2.77 [18:56:51] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.19, 3.24, 3.26 [18:58:40] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.33, 2.89, 2.71 [18:59:00] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 3.98, 3.58, 3.10 [18:59:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.51, 10.30, 10.12 [19:00:56] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 4.32, 3.82, 3.24 [19:01:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.21, 10.83, 10.32 [19:02:05] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.64, 4.32, 4.03 [19:02:51] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.41, 3.38, 3.15 [19:03:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.99, 11.01, 10.47 [19:04:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.52, 11.48, 10.03 [19:04:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.70, 3.96, 3.92 [19:05:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.71, 9.39, 9.94 [19:06:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.62, 11.73, 10.33 [19:08:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.38, 12.22, 10.67 [19:08:20] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.797 second response time [19:11:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.71, 11.00, 10.37 [19:12:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.64, 10.46, 10.42 [19:13:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.33, 3.00 [19:13:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.19, 10.92, 10.41 [19:14:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.14, 9.45, 10.06 [19:14:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.34, 3.60, 3.62 [19:15:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 2.96, 2.90 [19:17:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.21, 9.46, 9.95 [19:18:23] PROBLEM - gluster102 Disk Space on gluster102 is WARNING: DISK WARNING - free space: / 57742 MB (10% inode=72%); [19:18:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.40, 3.78, 3.74 [19:19:12] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=roboquestwiki --no-updates --username-prefix=wikia:roboquest /home/reception/ImportDump/roboquestwiki-20221111191411.xml (START) [19:19:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:49] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=roboquestwiki --no-updates --username-prefix=wikia:roboquest /home/reception/ImportDump/roboquestwiki-20221111191411.xml (END - exit=0) [19:22:50] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=roboquestwiki (START) [19:23:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:40] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=roboquestwiki (END - exit=0) [19:25:41] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=roboquestwiki --active --update (END - exit=0) [19:25:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:28] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.44, 3.20, 2.99 [19:28:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.68, 9.63, 9.38 [19:28:26] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.97, 3.05, 2.96 [19:29:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.98, 9.69, 9.59 [19:30:26] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.90, 3.05, 3.35 [19:30:33] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [19:31:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.60, 9.33, 9.47 [19:32:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.78, 9.83, 9.56 [19:33:42] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 5.12, 3.88, 3.16 [19:33:47] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:50] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 2.89, 3.53, 3.12 [19:36:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.88, 11.17, 10.17 [19:37:33] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.67, 3.21, 3.04 [19:37:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.090 second response time [19:38:04] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.261 second response time [19:38:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.06, 10.75, 10.12 [19:39:13] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.70, 3.70, 3.22 [19:40:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.38, 9.53, 9.76 [19:41:07] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.55, 3.56, 3.22 [19:41:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.84, 10.43, 9.82 [19:43:02] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.36, 3.63, 3.27 [19:44:09] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.86, 9.75, 9.71 [19:45:47] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.85, 3.14, 2.04 [19:46:11] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.30, 3.58, 3.14 [19:47:47] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.12, 2.85, 2.07 [19:48:13] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.271 second response time [19:48:46] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.77, 3.71, 3.51 [19:50:08] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.76, 3.12, 3.06 [19:50:40] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.80, 4.10, 3.67 [19:54:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.01, 3.43, 3.22 [19:54:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.44, 3.65, 3.58 [19:55:33] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [19:57:15] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.67, 3.32, 2.56 [19:58:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.29, 3.96, 3.68 [19:59:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.07, 2.56, 2.38 [20:00:00] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 1.46, 2.63, 2.97 [20:00:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.08, 3.82, 3.68 [20:03:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.14, 10.89, 9.91 [20:04:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.29, 3.01, 3.37 [20:04:49] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.30, 3.72, 3.29 [20:05:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.47, 10.50, 9.90 [20:06:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.42, 3.49, 3.25 [20:07:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.00, 9.67, 9.67 [20:08:41] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.47, 3.19, 3.17 [20:11:54] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:55] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.828 second response time [20:23:01] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.37, 3.62, 2.54 [20:23:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 4.00, 3.83, 3.25 [20:26:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [20:29:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.65, 3.31, 2.92 [20:29:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.23, 3.16, 3.16 [20:31:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [20:32:37] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:25] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.293 second response time [20:34:35] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.673 second response time [20:36:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.51, 4.40, 3.11 [20:37:40] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.89, 3.27, 2.66 [20:39:37] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.31, 3.05, 2.67 [20:39:50] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.91, 3.65, 3.27 [20:43:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.58, 3.47, 3.22 [20:44:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.76, 3.95, 3.36 [20:44:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.71, 2.63, 2.94 [20:45:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.46, 3.09, 3.11 [20:46:28] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.29, 3.36, 3.21 [20:48:06] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.041 second response time [20:49:30] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:19] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:26] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw132 [20:51:29] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.455 second response time [20:51:47] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw132 [20:52:21] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.134 second response time [20:52:26] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:53:25] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [20:53:48] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [20:55:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.83, 4.53, 3.21 [20:55:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.99, 3.83, 3.10 [20:59:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.66, 3.06, 2.90 [21:04:24] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:04:57] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.295 second response time [21:05:08] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:47] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 2 backends are down. mw121 mw122 [21:06:20] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw121 [21:06:23] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.107 second response time [21:06:50] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw121 [21:06:50] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:23] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 4 backends are down. mw121 mw131 mw132 mw141 [21:07:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 2.99, 2.52 [21:08:58] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.529 second response time [21:09:16] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.242 second response time [21:09:39] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.15, 2.73, 2.49 [21:10:35] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:09] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:29] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [21:12:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [21:13:06] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.344 second response time [21:17:26] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:19:06] PROBLEM - gluster102 PowerDNS Recursor on gluster102 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:19:20] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.472 second response time [21:19:23] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:19:25] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:20:30] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.658 second response time [21:21:08] RECOVERY - gluster102 PowerDNS Recursor on gluster102 is OK: DNS OK: 6.703 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [21:21:20] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.322 second response time [21:21:24] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.510 second response time [21:21:46] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:14] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.60, 3.09, 2.33 [21:23:28] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:46] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.724 second response time [21:23:50] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5936 bytes in 6.536 second response time [21:24:14] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.21, 2.60, 2.23 [21:25:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.72, 4.03, 3.03 [21:25:23] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.223 second response time [21:26:22] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: connect to address 2a10:6740::6:503 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:27:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.46, 3.28, 2.87 [21:27:25] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: connect to address 2a10:6740::6:309 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:27:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [21:27:52] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: connect to address 2a10:6740::6:404 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:28:00] PROBLEM - mw142 Check Gluster Clients on mw142 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [21:28:05] PROBLEM - mw131 Check Gluster Clients on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [21:28:07] PROBLEM - mw141 Check Gluster Clients on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [21:28:09] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [21:28:12] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: connect to address 2a10:6740::6:309 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:28:30] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [21:28:35] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: connect to address 2a10:6740::6:404 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:28:49] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [21:28:50] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:28:58] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.113 second response time [21:29:19] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [21:29:20] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 6% [21:29:31] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.323 second response time [21:29:35] RECOVERY - mw131 JobRunner Service on mw131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:29:40] RECOVERY - mw131 Check Gluster Clients on mw131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [21:29:45] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.571 second response time [21:29:47] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [21:29:47] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.588 second response time [21:29:47] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 1.602 second response time [21:29:50] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [21:29:52] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.582 second response time [21:29:57] !log reboot mw* [21:29:58] RECOVERY - mw142 Check Gluster Clients on mw142 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [21:30:06] RECOVERY - mw141 Check Gluster Clients on mw141 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [21:30:29] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [21:31:05] !log reboot mw* [21:31:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:13] !log restart varnish on cp* [21:31:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.22, 3.54, 2.68 [21:34:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.51, 3.95, 2.85 [21:35:49] !log MediaWiki outage started at 21:04 UTC lasting until 21:31 UTC [21:35:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.89, 5.10, 3.52 [21:38:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.93, 3.98, 3.17 [21:40:10] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.60, 4.38, 3.42 [21:40:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.76, 3.96, 3.42 [21:42:43] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.01, 3.23, 3.21 [21:44:13] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.60, 3.94, 3.46 [21:45:26] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.252 second response time [21:46:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.99, 3.97, 3.53 [21:46:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.97, 4.21, 3.61 [21:48:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 3.88, 4.13, 3.65 [21:50:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.89, 3.70, 3.56 [21:52:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.16, 3.74, 3.58 [21:54:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.26, 3.24, 3.42 [21:56:41] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.322 second response time [21:58:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.82, 10.33, 8.09 [21:58:14] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:26] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.28, 3.03, 3.31 [22:00:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.05, 8.80, 7.81 [22:00:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [22:02:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.66, 3.68, 3.85 [22:04:23] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.86, 4.21, 3.63 [22:04:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.90, 4.23, 4.01 [22:06:20] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.00, 3.79, 3.50 [22:06:40] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.291 second response time [22:08:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.88, 10.17, 8.81 [22:08:17] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.295 second response time [22:10:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [22:12:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.87, 9.72, 8.94 [22:18:14] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.61, 3.95, 4.00 [22:20:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.04, 4.05, 4.03 [22:20:45] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.78, 2.71, 2.00 [22:22:11] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.65, 3.53, 3.84 [22:22:45] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.97, 2.75, 2.09 [22:23:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.24, 3.19, 2.50 [22:25:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.95, 2.99, 2.51 [22:26:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.46, 4.13, 4.02 [22:30:05] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.30, 3.62, 3.88 [22:34:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 6.62, 4.52, 4.13 [22:34:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.67, 3.73, 3.05 [22:34:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.88, 3.61, 3.95 [22:34:37] [02puppet] 07MacFan4000 reviewed pull request 03#2996 commit - 13https://github.com/miraheze/puppet/pull/2996#discussion_r1020588997 [22:34:38] [url] allow direct pushing of certs to GitHub T7582 by Reception123 · Pull Request #2996 · miraheze/puppet · GitHub | github.com [22:34:40] [02puppet] 07MacFan4000 reviewed pull request 03#2996 commit - 13https://github.com/miraheze/puppet/pull/2996#discussion_r1020593254 [22:34:41] [url] allow direct pushing of certs to GitHub T7582 by Reception123 · Pull Request #2996 · miraheze/puppet · GitHub | github.com [22:34:41] [02puppet] 07MacFan4000 reviewed pull request 03#2996 commit - 13https://github.com/miraheze/puppet/pull/2996#discussion_r1020592663 [22:34:42] [url] allow direct pushing of certs to GitHub T7582 by Reception123 · Pull Request #2996 · miraheze/puppet · GitHub | github.com [22:34:44] [02puppet] 07MacFan4000 reviewed pull request 03#2996 commit - 13https://github.com/miraheze/puppet/pull/2996#discussion_r1020594965 [22:34:45] [url] allow direct pushing of certs to GitHub T7582 by Reception123 · Pull Request #2996 · miraheze/puppet · GitHub | github.com [22:34:46] [02puppet] 07MacFan4000 reviewed pull request 03#2996 commit - 13https://github.com/miraheze/puppet/pull/2996#discussion_r1020594725 [22:34:47] [url] allow direct pushing of certs to GitHub T7582 by Reception123 · Pull Request #2996 · miraheze/puppet · GitHub | github.com [22:34:47] [02puppet] 07MacFan4000 reviewed pull request 03#2996 commit - 13https://github.com/miraheze/puppet/pull/2996#discussion_r1020594132 [22:34:48] ... [22:34:49] [02puppet] 07MacFan4000 reviewed pull request 03#2996 commit - 13https://github.com/miraheze/puppet/pull/2996#discussion_r1020593783 [22:34:50] ... [22:34:50] [02puppet] 07MacFan4000 reviewed pull request 03#2996 commit - 13https://github.com/miraheze/puppet/pull/2996#discussion_r1020595064 [22:34:51] ... [22:35:37] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.55, 3.61, 2.90 [22:36:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.02, 3.45, 3.03 [22:37:34] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.54, 3.94, 3.10 [22:38:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.34, 3.10, 2.95 [22:38:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.80, 4.15, 4.05 [22:40:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.82, 3.63, 4.00 [22:40:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.91, 3.58, 3.85 [22:41:57] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.78, 3.65, 3.93 [22:42:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 7.26, 4.77, 4.36 [22:42:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.36, 3.98, 3.97 [22:46:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.08, 3.69, 3.85 [22:47:17] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.24, 3.80, 3.48 [22:50:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 5.48, 4.08, 3.92 [22:51:10] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.34, 3.88, 3.55 [22:51:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 5.31, 3.91, 3.74 [22:52:14] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.48, 2.80, 1.80 [22:53:07] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.86, 3.57, 3.48 [22:53:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.09, 3.59, 3.65 [22:54:14] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.05, 2.62, 1.87 [22:55:03] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 3.32, 3.36, 3.40 [22:57:54] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.68, 3.81, 3.70 [22:58:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.49, 3.63, 3.87 [22:59:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.81, 3.25, 2.61 [22:59:41] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.72, 3.51, 3.59 [23:00:56] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 2.54, 3.37, 3.44 [23:01:17] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.51, 3.76, 2.87 [23:01:41] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.102 second response time [23:01:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 4.60, 3.96, 3.74 [23:02:52] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.89, 3.15, 3.35 [23:03:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.98, 3.49, 2.86 [23:05:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.67, 2.77, 2.67 [23:08:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.15, 3.44, 3.63 [23:09:45] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 5.10, 3.97, 3.60 [23:09:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 2.77, 3.76, 3.82 [23:10:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.20, 3.48, 3.93 [23:11:41] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.86, 3.87, 3.61 [23:12:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.13, 3.90 [23:12:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.04, 3.61, 3.68 [23:14:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.61, 3.84, 3.94 [23:14:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.45, 3.92 [23:16:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.79, 3.90, 3.96 [23:16:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.38, 3.83 [23:18:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.19, 4.25, 4.09 [23:19:28] PROBLEM - gluster112 Current Load on gluster112 is CRITICAL: CRITICAL - load average: 4.48, 3.84, 3.60 [23:20:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.05, 3.99, 3.98 [23:20:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.59, 3.87 [23:20:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.59, 3.85, 3.69 [23:21:24] PROBLEM - gluster112 Current Load on gluster112 is WARNING: WARNING - load average: 3.79, 3.69, 3.57 [23:22:27] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.61, 3.91, 3.96 [23:22:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.00, 3.98, 3.95 [23:23:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.69, 2.99, 2.51 [23:23:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.20, 2.87, 3.36 [23:24:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.07, 3.88, 3.94 [23:24:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.35, 3.27, 3.69 [23:25:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.65, 3.82, 2.89 [23:26:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 2.73, 3.59, 3.64 [23:27:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.50, 3.63, 2.94 [23:27:14] RECOVERY - gluster112 Current Load on gluster112 is OK: OK - load average: 2.43, 3.10, 3.36 [23:27:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.34, 3.56, 3.56 [23:29:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.33, 2.81, 2.71 [23:30:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.93, 3.81, 3.97 [23:30:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.96, 2.60, 3.28 [23:34:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.87, 9.91, 8.87 [23:34:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.46, 3.48 [23:34:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.96, 3.03, 3.36 [23:36:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.79, 10.44, 9.21 [23:36:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 3.02, 3.32 [23:38:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.19, 9.43, 9.00 [23:40:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.87, 3.62 [23:40:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 4.82, 3.80, 3.54 [23:41:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.73, 3.24, 3.39 [23:42:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 5.09, 4.00, 3.88 [23:42:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 3.59, 3.67, 3.52 [23:42:48] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.45, 2.93, 2.36 [23:44:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 2.95, 3.70, 3.79 [23:44:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.00, 4.43, 3.85 [23:44:48] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.90, 2.88, 2.41 [23:46:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 4.29, 3.76, 3.79 [23:46:37] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.79, 3.28, 3.39 [23:48:26] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 3.10, 3.72, 3.78 [23:55:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.61, 2.86 [23:59:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.62, 2.93, 2.79